Skip to main content ↑ ↓
Topic: FreeRTOS (Read 25824 times) previous topic - next topic

Re: FreeRTOS

Reply #45 –
[quote author="mattcallow"]
...but memory is quite tight at the moment...[/quote]

This was one of the things that struck me as strange the first time I compiled the project but I have not looked in to it too far yet.  The uIP code seems to have a lot of (and many big) variables but I imagine this has already been scrutinized quite a bit by others already.  The other big target is the heap.  Do we need 3kB of heap space?  I have not looked yet but what requires this much dynamic memory allocation?

...

Okay, I just took a quick look and you are using heap_3.c in FreeRTOS.  I have always used heap_1.c before in my projects.  What drove the need for true dynamic memory?  Was it the uIP stack or the FatFs file system?

...

Okay, another quick look says they are both using it.  I do not think this is something I will get too soon but but my guess is that they do not need that much.  The amount needed by the file system should be very predictable based on how many files you need to have open at once (1 or 2?) and whether or not it is doing any buffering.  The uIP code I have no idea about what it should actually need.


-Eric

Re: FreeRTOS

Reply #46 –
How is this effort proceeding?  Is more help required?  I may have some time in a few weeks to help if it makes sense.

Re: FreeRTOS

Reply #47 –
[quote author="rhyde"]
How is this effort proceeding?
[/quote]

Well...

I guess I should start by saying I failed miserably...

...that is I failed miserably at not making major functional changes and wholesale format changes at the same time.

The more I went through the code the more I found things that weren't quite right or were just ugly.  (edit: I should have added "or were things that I just didn't like.")  Trying to hack fixes in would only have made the situation worse.  So I decided to forget about trying not to change things too much.  I also used a bunch of code I wrote for other projects that does initialization and what not.

So other than completely rearranging the source files and directory structure I have:
  • Fixed the oscillator PLL switching.  The original code was not unlocking the OSCCON low and high registers before writing them so the PLL was never changed from the reset default.  The defaults would have it running at over 92MHz (80MHz Max)  so I don't know if it was actually running overclocked or if it would somehow end up running slower since parts of the oscillator circuit may not have been able to keep up.  I added a couple files  (osc.c/h) I use for configuring the oscillator to make it cleaner.
  • The configuration bits were being set in two different files and were not quite correct.  For example, the OSC2/RA3 pin was not set as a digital pin as is needed by the new v1.1 hardware.  There may have been something other issue too -- I don't remember now.  Again, I added a file (configbits.h) I use to keep this clean.
  • I cleaned up the peripheral pin select setup and moved it to separate files (pps.c/h & ppsdef.h) that use standard macros to make the configuration a little easier and more obvious.
  • I implemented the check for whether a USB host is attached and the FTDI chip is powered before enabling the serial port transmitter.  There was a note about this in the source but it was not being done.  Right now it only checks one time on start up and turns it on or off permanently.   I will try to make this a dynamic check later.
  • The value of BRG was one count low.  It was 0.9% high before now it is 0.2% low.  Though either of these are well within spec.
  • I wrote a temporary replacement for the standard library write() function because the Microchip included version would always enable the UART transmitter.  If I have it off because there is no USB host attached I do not want a call to printf() to turn it on.  Eventually this will interface with the system logger.  It seemed cleaner and easier to make the change once in write() instead of changing each call to printf() for now.  Once I get to work on the logger I may want to do this differently but lots of code seems to just use a printf() to spit out debugging info so this might end up being the best solution anyway.
  • I added interrupt driven serial code with circular transmit and receive buffers.  This took a little longer than it should have because the routines I had been using in my projects were copied out of a book and so I thought I should rewrite them from scratch for this.  Every time I try to code a circular buffer I end up confusing myself and so have to work through the code with paper and a pencil to make sure all the index pointer math is correct.  I have versions of putc() and getc() implemented so far.
This is where I am at as of right this minute.  I am just now starting to write the console/terminal code to create the user interface.  Once I get this working with the serial port I will also connect it to the telnet server.

[quote author="rhyde"]
Is more help required?
[/quote]

Right now I don't know what is going to happen with the code I am working on.  Since Matt started this project I guess it is kind of his call what he does with it.  I am working on this because I know I will use it myself.  I will post it here once I get to a good stopping point and then people can look at it.  It will not be able to be merged back into SVN because it is too different.  Whether Matt wants to port any of my work to the SVN version is up to him.  If anyone else actually wants to use it I guess we could call it a fork and see if Ian wants to add it to SVN.  If Matt really wanted to I suppose he could use my version.  Otherwise it will just be a personal project.

[quote author="rhyde"]
I may have some time in a few weeks to help if it makes sense.
[/quote]

By that point I should have a version or two of mine posted here.  I don't know how much time Matt has to devote to this.  I know he wants to get the stability of the web server fixed.  That might be a good area to look at.  Any work done on the uIP code would be useful to everyone.  I don't expect to be mucking around in the uIP code if I can avoid it.  With the possible exception of trying to figure out how much dynamic memory it really needs and reducing the heap.


-Eric

Re: FreeRTOS

Reply #48 –
[quote author="Eric"]
Right now I don't know what is going to happen with the code I am working on.  Since Matt started this project I guess it is kind of his call what he does with it.  I am working on this because I know I will use it myself.  I will post it here once I get to a good stopping point and then people can look at it.  It will not be able to be merged back into SVN because it is too different.  Whether Matt wants to port any of my work to the SVN version is up to him.  If anyone else actually wants to use it I guess we could call it a fork and see if Ian wants to add it to SVN.  If Matt really wanted to I suppose he could use my version.  Otherwise it will just be a personal project.
[/quote]
All the code is open source, so people can change it as they wish. It seems a shame to create a fork of the existing code, but so be it.

[quote author="Eric"]
By that point I should have a version or two of mine posted here.  I don't know how much time Matt has to devote to this.  I know he wants to get the stability of the web server fixed.  That might be a good area to look at.  Any work done on the uIP code would be useful to everyone.  I don't expect to be mucking around in the uIP code if I can avoid it.  With the possible exception of trying to figure out how much dynamic memory it really needs and reducing the heap.
[/quote]

I think it may be possible to reduce the heap size. I don't know much is really needed. The reason for using heap_3.c rather than either of the other heap models is an effort to consolidate the dynamic memory allocation schemes in use, which (i think) will lead to better use of the available memory. Currently, the microchip libraries (e.g. printf) will use dynamic memory (i assume via malloc/free), uIP has it's own memory blocks (used in the telnet server), FreeRTOS needs to use one of heap_1/2/3.c and FatFS will need dynamic memory for LFN. Using the heap as the underlying dynamic memory pool for all these schemes seemed like the best way to go (in terms of memory usage).

Matt

Re: FreeRTOS

Reply #49 –
[quote author="mattcallow"]
It seems a shame to create a fork of the existing code, but so be it.  [/quote]

Agreed.  I do not necessarily want to create a fork (I just threw it out as an option) which is why I am just going to call this a personal project and offer anything I write back to the forum.  I guess all I am really saying is that I do not want to try to back-port (or whatever it's called) my changes in to the SVN version because my code is now so different.  If you want to merge any of my stuff in great, if not don't worry about it.  If you like what I end up with enough you could possibly even switch the code in SVN to it if you felt so inclined.  I have never used SVN other than for just downloading code so I don't know how it would deal with the large structural changes I've made.

[quote author="mattcallow"]
I think it may be possible to reduce the heap size. I don't know much is really needed. The reason for using heap_3.c rather than either of the other heap models is an effort to consolidate the dynamic memory allocation schemes in use, which (i think) will lead to better use of the available memory. Currently, the microchip libraries (e.g. printf) will use dynamic memory (i assume via malloc/free), uIP has it's own memory blocks (used in the telnet server), FreeRTOS needs to use one of heap_1/2/3.c and FatFS will need dynamic memory for LFN. Using the heap as the underlying dynamic memory pool for all these schemes seemed like the best way to go (in terms of memory usage).[/quote]

In the last couple hours I started bumping up against the the memory limit so I began investigating the issue.  It turns out uIP does not use any dynamic memory (as you noted above).  It statically declares all the buffers it needs at compile time.  FatFs also does not use dynamic memory unless long file name (LFN) support is turned (as you also noted).  Even with LFN support enabled it looks like it has an option to just use the stack.  So FatFs my never need dynamic memory.  If it does you need to write its ff_malloc() for it anyway.  I switched FreeRTOS to heap_1.c (which is what I have always used) since FreeRTOS never frees memory unless you delete a task or a queue -- and I can't think of a case where you would ever want to do this.  This let me set the compiler heap to zero.  You can't disable it or the linker will complain if you use any stdio.h input or output functions.  But if you never need a FILE * pointer (like if you only use printf() to a serial port) you can set the heap to zero.

I also noticed that the the configMINIMAL_STACK_SIZE was set to 256 bytes and the uIP task was using twice this value.  This is probably twice what most tasks need but it turns out it was a good value for the uIP task.  Was this a guess or did you measure it?  I set the configMINIMAL_STACK_SIZE back down to 115, the default for the dsPIC/PIC24 port, but created a configUIP_STACK_SIZE of 512 for the uIP task.  I measured the uIP stack grow to a max of 498 so 512 is probably a good value for now.  We'll need to keep an eye on this and see if it gets any closer to the top of the stack.  With the configTOTAL_HEAP_SIZE set at 2048 and the uIP task and two other tasks (the idle task and my console task) of default stack size there are still 310 bytes left in the FreeRTOS heap -- probably enough for two more normal sized tasks.  Of course the configTOTAL_HEAP_SIZE could just be brought down to make this go to zero if desired since once all tasks and all queues are created no more memory is ever needed.

All this brought the total memory usage down to about 79% from what ever it was before -- ninety-something I think.

Now that I have some breathing room I'll get back to what I was working on.


-Eric

Re: FreeRTOS

Reply #50 –
Eric,

Sounds like a ton of progress. I can't help but make code my own either, especially if I want to understand it thoroughly.

Let's get you SVN access so you can keep your progress in a separate folder or something. I'm sure there's other people (me :) ) who would like to play along at home

Here's the info about getting a password for Google Code:
http://dangerousprototypes.com/docs/Using_SVN
Got a question? Please ask in the forum for the fastest answers.

Re: FreeRTOS

Reply #51 –
[quote author="Eric"]
In the last couple hours I started bumping up against the the memory limit so I began investigating the issue.  It turns out uIP does not use any dynamic memory (as you noted above).  It statically declares all the buffers it needs at compile time.  FatFs also does not use dynamic memory unless long file name (LFN) support is turned (as you also noted).  Even with LFN support enabled it looks like it has an option to just use the stack.  So FatFs my never need dynamic memory.  If it does you need to write its ff_malloc() for it anyway.  I switched FreeRTOS to heap_1.c (which is what I have always used) since FreeRTOS never frees memory unless you delete a task or a queue -- and I can't think of a case where you would ever want to do this.  This let me set the compiler heap to zero.  You can't disable it or the linker will complain if you use any stdio.h input or output functions.  But if you never need a FILE * pointer (like if you only use printf() to a serial port) you can set the heap to zero.
[/quote]
Good to know that printf doesn't need the heap for serial port output. I'm still not sure about using the stack for LFN, since that would require a large stack for each task using FatFS. Anyway, currently it doesn't matter since LFN is disabled.

[quote author="Eric"]
I also noticed that the the configMINIMAL_STACK_SIZE was set to 256 bytes and the uIP task was using twice this value.  This is probably twice what most tasks need but it turns out it was a good value for the uIP task.  Was this a guess or did you measure it?  I set the configMINIMAL_STACK_SIZE back down to 115, the default for the dsPIC/PIC24 port, but created a configUIP_STACK_SIZE of 512 for the uIP task.  I measured the uIP stack grow to a max of 498 so 512 is probably a good value for now.  We'll need to keep an eye on this and see if it gets any closer to the top of the stack.  With the configTOTAL_HEAP_SIZE set at 2048 and the uIP task and two other tasks (the idle task and my console task) of default stack size there are still 310 bytes left in the FreeRTOS heap -- probably enough for two more normal sized tasks.  Of course the configTOTAL_HEAP_SIZE could just be brought down to make this go to zero if desired since once all tasks and all queues are created no more memory is ever needed.
[/quote]
Stack size was just a guess. I was, at one point, going to put the uIP code in the idle task. In that case, i needed to increase  configMINIMAL_STACK_SIZE, because that's the only way (i know) of changing the stack size of the idle task. Since uIP is running in it's own task, the change is not required.

Maybe once you've posted your code, I'll try some of this to see if it makes the stack more stable.

Matt

Re: FreeRTOS

Reply #52 –
[quote author="ian"]
Sounds like a ton of progress.[/quote]

Thanks.  I finally got in the programming "groove" and have started to move forward now.

[quote author="ian"]
I can't help but make code my own either, especially if I want to understand it thoroughly. [/quote]

Thanks for understanding.  It was certainly never my intent to create division and I owe Matt a debt of gratitude for getting this thing kicked off.  I don't want all the changes I have made to come across in any way negative.  Matt did the real hard work here of getting this thing going.  He got all the pieces together and functional which is where I tend to get bogged down.  Now that it is running it is easy for me to make incremental improvements which I am much better at.

[quote author="ian"]
Let's get you SVN access so you can keep your progress in a separate folder or something. ... Here's the info...[/quote]

I'll read up on it and try to learn how to use SVN.

[quote author="ian"]
I'm sure there's other people (me :) ) who would like to play along at home[/quote]

I'm not quite there yet.  There are still some things I want to go through first because they will probably cause major structural changes.  Like I just noticed that the SD code is setting up Time 1 but Timer 1 is then later configured by the FreeRTOS scheduler when it starts.  So right now there are two different initializations for Timer 1 in two different sections of code.  I need to go through this and figure out what FatFs needs.  I will probably move them on to different timers.  I have done this before with FreeRTOS, moved it to Timer 5 because Timer 1 is used by certain hardware peripherals like the ADC.  Then the scheduler "tick" can be decoupled from the file system timing and appropriate intervals can be set for each independently.

So till I get through a few more things like this there is still likely to be large code changes. 

I also need to figure out what to do about another licensing issue.  My uart.c file has a lot of code in it from the Microchip peripheral libraries that come with the C30 compiler.  Some time ago I took the relevant library functions, which were each in their own file, and consolidated them in to one file with some code I added.  The library routines were written for the most general case (like the ability to deal with 9 bit data) that I knew I would never need so I cleaned them up a bit.  Since everyone with the C30 compiler already has the peripheral libraries I can probably just pull the library functions back out of my file.  The only downside is that you are back to the bulky generic versions -- that and I like to have ALL the relevant source in the project directory where I can see and touch it -- it also really bugs me that Microchip does not include the source for the C standard library they include.  The other option is for me to rewrite the file from scratch.  I have always just written code for myself before and so never worried about copyright or licensing issues.  Any thoughts on this?

[quote author="mattcallow"]
I'm still not sure about using the stack for LFN, since that would require a large stack for each task using FatFS. [/quote]

I'm not really sure yet either.  We'll have to figure that out when we get to it.  Since we have to write the ff_malloc() and free commands anyway we have a lot of flexibility.

[quote author="mattcallow"]
Stack size was just a guess. [/quote]

Good guess then.

[quote author="mattcallow"]
Maybe once you've posted your code, I'll try some of this to see if it makes the stack more stable.[/quote]

That would be great.

I also found there was a project setting somewhere with a flag something like _ICD2RAM = 1 or similar.  Removing that also freed up some more memory.  I don't know if you have an ICD2 or if that was left over from somewhere else.  I think this flag should normally be enabled and disabled by MPLAB as you turn debugging on and off.  I have an ICD3 and not an ICD2 so my MPLAB may not have known what that flag was for and assumed it was a user flag.


-Eric

Re: FreeRTOS

Reply #53 –
[quote author="Eric"]
...Like I just noticed that the SD code is setting up Time 1 but Timer 1 is then later configured by the FreeRTOS scheduler when it starts.  So right now there are two different initializations for Timer 1 in two different sections of code.  I need to go through this and figure out what FatFs needs.  I will probably move them on to different timers.  I have done this before with FreeRTOS, moved it to Timer 5 because Timer 1 is used by certain hardware peripherals like the ADC.  Then the scheduler "tick" can be decoupled from the file system timing and appropriate intervals can be set for each independently.
[/quote]
This is because I initialise the SD card before the scheduler is started. Because the code to init the SD card is not thread safe.
This is really a hack.  As I mentioned earlier, probably the best way is to make the SD init code thread safe, add a 'initialised' variable, and allow it to be called from any thread.

[quote author="Eric"]
I also found there was a project setting somewhere with a flag something like _ICD2RAM = 1 or similar.  Removing that also freed up some more memory.  I don't know if you have an ICD2 or if that was left over from somewhere else.  I think this flag should normally be enabled and disabled by MPLAB as you turn debugging on and off.  I have an ICD3 and not an ICD2 so my MPLAB may not have known what that flag was for and assumed it was a user flag.
[/quote]
I only have pickit2. I expect that is a hangover from the original FreeRTOS port.

Matt

Re: FreeRTOS

Reply #54 –
[quote author="mattcallow"]
[quote author="Eric"]
...I just noticed that the SD code is setting up Time 1 but Timer 1 is then later configured by the FreeRTOS scheduler...[/quote]
This is because I initialise the SD card before the scheduler is started. Because the code to init the SD card is not thread safe.
This is really a hack.  As I mentioned earlier, probably the best way is to make the SD init code thread safe, add a 'initialised' variable, and allow it to be called from any thread. [/quote]

Good to know, thanks.  I'll keep that in mind.

[quote author="mattcallow"]
[quote author="Eric"]
I also found there was a project setting somewhere with a flag something like _ICD2RAM = 1 or similar.  Removing that also freed up some more memory.[/quote]
I only have pickit2. I expect that is a hangover from the original FreeRTOS port.[/quote]

It wasn't much but every little bit helps -- especially when it was just being wasted.

-Eric

Re: FreeRTOS

Reply #55 –
[quote author="Eric"]... I just noticed that the SD code is setting up Time 1 but Timer 1 is then later configured by the FreeRTOS scheduler when it starts.  So right now there are two different initializations for Timer 1 in two different sections of code.  I need to go through this and figure out what FatFs needs.  I will probably move them on to different timers.  I have done this before with FreeRTOS, moved it to Timer 5 because Timer 1 is used by certain hardware peripherals like the ADC.  Then the scheduler "tick" can be decoupled from the file system timing and appropriate intervals can be set for each independently.

So till I get through a few more things like this there is still likely to be large code changes.[/quote]Matt may have already answered this for the specific case at hand, but I wanted to mention a general technique.  First of all, the PIC usually has a lot of timers, so if you can use Timer 5 and get away with it, then you're done.  But sometimes you still end up running out of timers.  What I do in that case is find a faster timer which is a reasonable multiple of another timer, and then create a virtual timer.  Just use a volatile global variable to count the faster timer interrupts and then call the slower timer function as if a virtual timer had fired.  The scheduler "tick" is probably not something which has to run at a precise speed - unlike PWM or a serial port - and so the scheduler tick is a good candidate for a virtual timer.  You might already know these tricks, but I thought I'd pass them on anyway.

Re: FreeRTOS

Reply #56 –
[quote author="rsdio"]
First of all, the PIC usually has a lot of timers, so if you can use Timer 5 and get away with it, then you're done.  But sometimes you still end up running out of timers.  What I do in that case is find a faster timer which is a reasonable multiple of another timer, and then create a virtual timer.  Just use a volatile global variable to count the faster timer interrupts and then call the slower timer function as if a virtual timer had fired.  The scheduler "tick" is probably not something which has to run at a precise speed - unlike PWM or a serial port - and so the scheduler tick is a good candidate for a virtual timer.  You might already know these tricks, but I thought I'd pass them on anyway.
[/quote]

I have never run out of timers before but I'll keep this in mind.  Usually it is just an issue of a particular peripheral needing to use a specific timer that necessitates moving other things around.

-Eric
 

Re: FreeRTOS

Reply #57 –
[quote author="Eric"]
I also need to figure out what to do about another licensing issue.  My uart.c file has a lot of code in it from the Microchip peripheral libraries that come with the C30 compiler.
[/quote]

I rewrote all my serial code now from scratch so there is no more Microchip peripheral library source code in it.

This gets me one step closer to being able to commit this to SVN.  Hopefully this weekend I will have it ready.

-Eric

Re: FreeRTOS

Reply #58 –
hi :)

noob with microchip PIC but i did few realtime systems long time ago...
so im trying first to compiling FreeRTOS from the trunk and i got linker error

Executing: "C:Program FilesMicrochipmplabc30v3.25binpic30-gcc.exe" -mcpu=33FJ128GP204 "main.o" "rtcc.o" "enc28j60.o" "nic_dma.o" "delay.o" "nic.o" "uip_task.o" "ff.o" "heap_3.o" "croutine.o" "list.o" "uip.o" "uip_arp.o" "timer.o" "portasm_dsPIC.o" "mmc.o" "memb.o" "port.o" "queue.o" "psock.o" "tasks.o" "httpd.o" "httpd-cgi.o" "http-strings.o" "dhcpc.o" "logger.o" -o"WebPlatform.cof" -Wl,--script="p33FJ128GP204.gld",--defsym=__MPLAB_BUILD=1,--defsym=__MPLAB_DEBUG=1,--heap=3096,--stack=512,--defsym=__ICD2RAM=1,-Map="WebPlatform.map"
heap: Link Error: Could not allocate section .heap, size = 3096 bytes, attributes = heap
Link Error: Could not allocate data memory


so ...  someone has a hint for me ?  :)

Re: FreeRTOS

Reply #59 –
[quote author="voidptr"]
hi :)

noob with microchip PIC but i did few realtime systems long time ago...
so im trying first to compiling FreeRTOS from the trunk and i got linker error

Executing: "C:Program FilesMicrochipmplabc30v3.25binpic30-gcc.exe" -mcpu=33FJ128GP204 "main.o" "rtcc.o" "enc28j60.o" "nic_dma.o" "delay.o" "nic.o" "uip_task.o" "ff.o" "heap_3.o" "croutine.o" "list.o" "uip.o" "uip_arp.o" "timer.o" "portasm_dsPIC.o" "mmc.o" "memb.o" "port.o" "queue.o" "psock.o" "tasks.o" "httpd.o" "httpd-cgi.o" "http-strings.o" "dhcpc.o" "logger.o" -o"WebPlatform.cof" -Wl,--script="p33FJ128GP204.gld",--defsym=__MPLAB_BUILD=1,--defsym=__MPLAB_DEBUG=1,--heap=3096,--stack=512,--defsym=__ICD2RAM=1,-Map="WebPlatform.map"
heap: Link Error: Could not allocate section .heap, size = 3096 bytes, attributes = heap
Link Error: Could not allocate data memory


so ...  someone has a hint for me ?  :)
[/quote]
The memory usage is not good at the moment. Try reducing the heap size a little (somewhere on the linker tab in the IDE I think)
Eric has also pointed out that you can remove the __ICD2RAM=1 symbol

Matt