I've managed to get FreeRTOS running on the Web platform.
Got a few LEDs blinking, and have started to integrate the uIP stack. I now have ping working.
I've used lots of code from the uIP stack in svn, but I've taken the latest version of uIP (1.0). I'm not sure what version was used in the uIP stack?
uIP v1.0 seems to include:
- http server
- dhcp client
- telnet server
So I will try to build those in next.
Matt
Outstanding! By the time I get my Web Platform you will have all the hard work done.
-Eric
Nice work! Please let me know when you have a link to the source, I'll post it up on the blog.
We started with a 0.9 AVR uIP port previously. 1.0 sounds much improved.
I think it is better for me to start using that Web Platform. I didn't do anything since I moved to a new home (dormitory actually, had one ethernet connection), but as I have an ethernet switch now, I can use that.
I have the telnet and dhcp examples working now. Although the telnet server doesn't do anything useful yet.
The next step if the web server. For that I need a file system. I've seen posts about DOSFS and FATFS in the forums. Any consensus on which is best?
Matt
I think FATFS is considered the best available now:
http://elm-chan.org/fsw/ff/00index_e.html (http://elm-chan.org/fsw/ff/00index_e.html)
I got the http server demo working last night. But there are some problems. It seems that the server 'hangs' when sending segments == mss.
If I reduce the size of the max segment sent, it all works OK. I need to work out why...
I haven't looked at using the SD card yet.
I also need to get the http and telnet servers working at the same time.
The http server has a simple scripting mechanism, which I think could easly be extended to control/report the pins on the PIC.
Currently, I'm not sure how useful it is using RTOS and uip, since uip runs in a single RTOS thread, so all network stuff must be done in that thread.
Matt
[quote author="mattcallow"]
Currently, I'm not sure how useful it is using RTOS and uip, since uip runs in a single RTOS thread, so all network stuff must be done in that thread.[/quote]
The key benefit of using FreeRTOS is when you want to start adding in completely unrelated application code. The "web" part is well partitioned running in the background requiring little thought because it just "works" with little to no dependence on what else is added.
Theoretically you could take the Bus Pirate code, run it is a separate task, tie its serial terminal IO to the telnet server and have a web enabled Bus Pirate in short order. The real beauty is being able to use the Web Platform to "web enable" another project. At least that's how I see it.
[quote author="mattcallow"]
I haven't looked at using the SD card yet.
[/quote]
This is where I would want to start looking at partitioning: network stack in one thread, file IO in another, serial console in another if used, application code in another thread, graphic LCD driver in another thread, possibly button scanning in another thread, etc. The FreeRTOS website has a design tutorial (http://http://www.freertos.org/tutorial/index.html) that walks through the trade-offs of various implementations of a design and talks about these issues.
Thanks again, keep up the great work.
-Eric
[quote author="Eric"]
The key benefit of using FreeRTOS is when you want to start adding in completely unrelated application code. The "web" part is well partitioned running in the background requiring little thought because it just "works" with little to no dependence on what else is added.
[/quote]
I think you've probably articulated my thoughts on this! It's the way I could see it working, just wasn't sure if others would find it useful.
This is different to the current uip stack port in svn, which allows multiple (co-operative) tasks to use the uip stack. I did contemplate that approach, but I think if we can manage with all network stuff in a single task, the code will be simpler.
[quote author="Eric"]
Theoretically you could take the Bus Pirate code, run it is a separate task, tie its serial terminal IO to the telnet server and have a web enabled Bus Pirate in short order. The real beauty is being able to use the Web Platform to "web enable" another project. At least that's how I see it.
[/quote]
That's an interesting idea. Are you talking about using 2 bits of hardware here (Web platform + bus pirate) or running the bus pirate code in the web platform?
[quote author="Eric"]
This is where I would want to start looking at partitioning: network stack in one thread, file IO in another, serial console in another if used, application code in another thread, graphic LCD driver in another thread, possibly button scanning in another thread, etc. The FreeRTOS website has a design tutorial (http://http://www.freertos.org/tutorial/index.html) that walks through the trade-offs of various implementations of a design and talks about these issues.
[/quote]
Yes, I've seen the tutorial. I read it a while ago when I first started to look at FreeRTOS.
I would agree with your separation of tasks, except possibly the file IO. I'm not sure (yet) if that would be better as a separate task or allow multiple tasks to access the file system and use a mutex to control access.
[quote author="Eric"]
Thanks again, keep up the great work.
[/quote]
Thanks for the encouragement
Matt
[quote author="mattcallow"]
[quote author="Eric"]
The key benefit of using FreeRTOS is when you want to start adding in completely unrelated application code. The "web" part is well partitioned running in the background requiring little thought because it just "works" with little to no dependence on what else is added.
[/quote]
I think you've probably articulated my thoughts on this! It's the way I could see it working, just wasn't sure if others would find it useful. [/quote]
Well at least that makes two of us on the same page.
[quote author="mattcallow"]
This is different to the current uip stack port in svn, which allows multiple (co-operative) tasks to use the uip stack. I did contemplate that approach, but I think if we can manage with all network stuff in a single task, the code will be simpler.[/quote]
Since TCP/IP wasn't in the PIC24/dsPIC port of FreeRTOS I have not looked at it at all. Which SVN are you referring to, FreeRTOS, uIP or some other?
[quote author="mattcallow"]
[quote author="Eric"]
Theoretically you could take the Bus Pirate code, run it is a separate task, tie its serial terminal IO to the telnet server and have a web enabled Bus Pirate in short order. The real beauty is being able to use the Web Platform to "web enable" another project. At least that's how I see it.
[/quote]
That's an interesting idea. Are you talking about using 2 bits of hardware here (Web platform + bus pirate) or running the bus pirate code in the web platform?
[/quote]
Running the Bus Pirate code on the Web Platform. The Bus Pirate with all its bit-banging timing loops may not be the best test case for this though. This was just a "for example." However, for one or two of the modes that use the PIC hardware peripherals it should not be too hard to do. I ripped a serial terminal out of one project and used FreeRTOS to graft it into another project which had only been using an LCD display. This is where an OS like FreeRTOS starts to shine.
[quote author="mattcallow"]
I would agree with your separation of tasks, except possibly the file IO. I'm not sure (yet) if that would be better as a separate task or allow multiple tasks to access the file system and use a mutex to control access.[/quote]
I would want the file I/O separate because the web server might not be the only process that needs to read or save files. I could easily see other tasks wanting to read config settings or save data or log files. As long as two threads did not try to open the same file RW it should not be a problem (other than memory limitations) to let different tasks have files open at the same time. I have not even though about how to make this work but I'm sure it has already been figured out somewhere.
Ordering a Web Platform is on my to-do list for tomorrow! Than maybe I can start helping a little -- or at least with testing.
-Eric
[quote author="Eric"]Running the Bus Pirate code on the Web Platform. The Bus Pirate with all its bit-banging timing loops may not be the best test case for this though. This was just a "for example." However, for one or two of the modes that use the PIC hardware peripherals it should not be too hard to do. I ripped a serial terminal out of one project and used FreeRTOS to graft it into another project which had only been using an LCD display. This is where an OS like FreeRTOS starts to shine.[/quote]Combining Bus Pirate features with a Web Platform would require optimized hardware interrupt routines, almost certainly written in PIC assembly. But, if you are successful in keeping the hardware stuff from taking over the CPU completely, then there's certainly room for a web service running in the background.
[quote author="Eric"]
[quote author="mattcallow"]
This is different to the current uip stack port in svn, which allows multiple (co-operative) tasks to use the uip stack. I did contemplate that approach, but I think if we can manage with all network stuff in a single task, the code will be simpler.[/quote]
Since TCP/IP wasn't in the PIC24/dsPIC port of FreeRTOS I have not looked at it at all. Which SVN are you referring to, FreeRTOS, uIP or some other?
[/quote]
This one: http://code.google.com/p/dangerous-prototypes-open-hardware/source/browse/#svn/trunk/web-platform/firmware/uIPStack (http://http://code.google.com/p/dangerous-prototypes-open-hardware/source/browse/#svn/trunk/web-platform/firmware/uIPStack)
[quote author="Eric"]
[quote author="mattcallow"]
[quote author="Eric"]
Theoretically you could take the Bus Pirate code, run it is a separate task, tie its serial terminal IO to the telnet server and have a web enabled Bus Pirate in short order. The real beauty is being able to use the Web Platform to "web enable" another project. At least that's how I see it.
[/quote]
That's an interesting idea. Are you talking about using 2 bits of hardware here (Web platform + bus pirate) or running the bus pirate code in the web platform?
[/quote]
Running the Bus Pirate code on the Web Platform. The Bus Pirate with all its bit-banging timing loops may not be the best test case for this though. This was just a "for example." However, for one or two of the modes that use the PIC hardware peripherals it should not be too hard to do. I ripped a serial terminal out of one project and used FreeRTOS to graft it into another project which had only been using an LCD display. This is where an OS like FreeRTOS starts to shine.
[/quote]
As you say, probably OK when using hardware peripherals. May be issues it the code relies on exact timing loops.
[quote author="Eric"]
[quote author="mattcallow"]
I would agree with your separation of tasks, except possibly the file IO. I'm not sure (yet) if that would be better as a separate task or allow multiple tasks to access the file system and use a mutex to control access.[/quote]
I would want the file I/O separate because the web server might not be the only process that needs to read or save files. I could easily see other tasks wanting to read config settings or save data or log files. As long as two threads did not try to open the same file RW it should not be a problem (other than memory limitations) to let different tasks have files open at the same time. I have not even though about how to make this work but I'm sure it has already been figured out somewhere.
[/quote]
I agree that multiple tasks should probably be able to access the file system. I'm just not sure that a separate task to handle file IO is the best way to handle that. I'll probably have a better idea once I start looking at the FatFS code.
[quote author="Eric"]
Ordering a Web Platform is on my to-do list for tomorrow! Than maybe I can start helping a little -- or at least with testing.
[/quote]
Great! I may have some useable code published by the time you get yours.
[quote author="rsdio"]
[quote author="Eric"]Running the Bus Pirate code on the Web Platform. The Bus Pirate with all its bit-banging timing loops may not be the best test case for this though. This was just a "for example." However, for one or two of the modes that use the PIC hardware peripherals it should not be too hard to do. I ripped a serial terminal out of one project and used FreeRTOS to graft it into another project which had only been using an LCD display. This is where an OS like FreeRTOS starts to shine.[/quote]Combining Bus Pirate features with a Web Platform would require optimized hardware interrupt routines, almost certainly written in PIC assembly. But, if you are successful in keeping the hardware stuff from taking over the CPU completely, then there's certainly room for a web service running in the background.
[/quote]
As I noted, the Bus Pirate may not have been the best example. That said, for the non-bit-banged protocols that use the hardware peripherals like SPI or UART I do not think it would be that hard or require assembly. I would definitely want to use interrupts to service the peripherals but FreeRTOS is designed to handle this. The ISR usually moves data between a circular buffer and the peripheral or, if the peripheral has a deep enough FIFO buffer, it just wakes up the process to service the peripheral directly.
The only thing that is usually highly incompatible with FreeRTOS, or any RTOS for that matter, are delay loops. These either need to use the OS API for longer delays or use interrupts for shorter delays. Hence why the bit-banged modes would not work well.
-Eric
[quote author="Eric"]for the non-bit-banged protocols that use the hardware peripherals like SPI or UART I do not think it would be that hard or require assembly. I would definitely want to use interrupts to service the peripherals but FreeRTOS is designed to handle this. The ISR usually moves data between a circular buffer and the peripheral or, if the peripheral has a deep enough FIFO buffer, it just wakes up the process to service the peripheral directly.[/quote]I assume that FreeRTOS already has PIC assembly for the interrupt routines. If not, they certainly need some specific platform code for the PIC interrupts, because I don't think that is very portable. In other words, even FreeRTOS would benefit from optimized PIC assembly - sometimes you can cut the cycles spent in interrupts to half or less using hand-coded assembly.
The only thing that is usually highly incompatible with FreeRTOS, or any RTOS for that matter, are delay loops. These either need to use the OS API for longer delays or use interrupts for shorter delays. Hence why the bit-banged modes would not work well.
Depending upon the clock rates of the big-banged protocols, converting delay loop code to Timer/counter code might improve accuracy and CPU utilization. But there are limits.
[quote author="rsdio"]
[quote author="Eric"]for the non-bit-banged protocols that use the hardware peripherals like SPI or UART I do not think it would be that hard or require assembly. I would definitely want to use interrupts to service the peripherals but FreeRTOS is designed to handle this. The ISR usually moves data between a circular buffer and the peripheral or, if the peripheral has a deep enough FIFO buffer, it just wakes up the process to service the peripheral directly.[/quote]I assume that FreeRTOS already has PIC assembly for the interrupt routines. If not, they certainly need some specific platform code for the PIC interrupts, because I don't think that is very portable. In other words, even FreeRTOS would benefit from optimized PIC assembly - sometimes you can cut the cycles spent in interrupts to half or less using hand-coded assembly.
[/quote]
As far as I know, FreeRTOS only uses 1 interrupt to provide a timer tick and context switching (in pre-emptive mode). This ISR is written in C.
Of course, you can write your own interrupt routines in assembly or C as you wish.
The kernel tick interrupt is set at the lowest priority, so that the kernel does not interfere with other interrupts.
[quote author="mattcallow"]As far as I know, FreeRTOS only uses 1 interrupt to provide a timer tick and context switching (in pre-emptive mode). This ISR is written in C.
Of course, you can write your own interrupt routines in assembly or C as you wish.
The kernel tick interrupt is set at the lowest priority, so that the kernel does not interfere with other interrupts.[/quote]I hope I'm not over-optimizing here, but the PIC has the ability to enable two interrupts, high priority and low priority. Each peripheral can be connected to either interrupt callback by setting its priority. This is the case for PIC18, but I haven't checked whether it's the same for PIC24. Seems like FreeRTOS should pick a Timer and assign it to low priority after enabling the two-tiered interrupt priority feature.
Might be worth writing the Bus Pirate interrupts in assembly, then calling the C interrupt function from the low priority interrupt after checking the Timer flag.
Interesting stuff. I'd like to do a net-enabled TCPIP project for Halloween (need to get to an auto-parts store first...), do you think this would be a good candidate? I planned to use the uIP stack (raw) or Microchip stack from past projects, but FreeRTOS would be fun.
[quote author="rsdio"]
[quote author="mattcallow"]As far as I know, FreeRTOS only uses 1 interrupt to provide a timer tick and context switching (in pre-emptive mode). This ISR is written in C.
Of course, you can write your own interrupt routines in assembly or C as you wish.
The kernel tick interrupt is set at the lowest priority, so that the kernel does not interfere with other interrupts.[/quote]I hope I'm not over-optimizing here, but the PIC has the ability to enable two interrupts, high priority and low priority. Each peripheral can be connected to either interrupt callback by setting its priority. This is the case for PIC18, but I haven't checked whether it's the same for PIC24. Seems like FreeRTOS should pick a Timer and assign it to low priority after enabling the two-tiered interrupt priority feature.
[/quote]
The dsPIC used in the web platform has 8 interrupt priority levels (well 7 really - unless you count disabled as a priority level). The FreeRTOS kernel already uses the lowest level for the tick timer (timer 1 in this case).
[quote author="ian"]
Interesting stuff. I'd like to do a net-enabled TCPIP project for Halloween (need to get to an auto-parts store first...), do you think this would be a good candidate? I planned to use the uIP stack (raw) or Microchip stack from past projects, but FreeRTOS would be fun.
[/quote]
Depends on what you want the project to do (and which PIC you choose)! From what I've seen of FreeRTOS, it seems pretty good. It should be possible to build up some useful libraries to handle stuff like network, file, serial, lcd etc which can be re-used on several projects. But there is obviously an overhead with using it, so if flash memory or RAM are tight then it may not be the best option.
[quote author="mattcallow"]
[quote author="ian"]
Interesting stuff. I'd like to do a net-enabled TCPIP project for Halloween (need to get to an auto-parts store first...), do you think this would be a good candidate? I planned to use the uIP stack (raw) or Microchip stack from past projects, but FreeRTOS would be fun.
[/quote]
Depends on what you want the project to do (and which PIC you choose)! From what I've seen of FreeRTOS, it seems pretty good. It should be possible to build up some useful libraries to handle stuff like network, file, serial, lcd etc which can be re-used on several projects. But there is obviously an overhead with using it, so if flash memory or RAM are tight then it may not be the best option.
[/quote]
Exactly. I am not an expert by any means but, I have found it far easier to reuse code when using FreeRTOS. When using typical supper-loop programming there are so many more interactions that I have never been very successful with code reuse. Code usually needs so many minor changes to tie it in that when I am done it is more different than the same.
But, as you said there is some overhead associated with this convenience. Like everything in engineering there are trade-offs. When I was using the mid range PICs I wrote exclusively in assembly so the move to the PIC24/dsPIC which brought a move to c seemed very wasteful and inefficient. However, I can write things with c that I probably never would have finished if I tried writing it in assembly. Similarly, the move to FreeRTOS brought about the ability to easily implement so many things that I probably never would have completed because of getting bogged down in the interdependencies. So for me the overhead is acceptable.
I would love to see a library of FreeRTOS tasks that can be included in a project to add specific functionality. I would be glad to donate any modules I write. At the moment the only one I can think of that I wrote from scratch is for a graphic LCD. The serial console I am using is based on code from a book so I would have to check on the license of that code. I would have to look but I do not think i have any other ones that would be generally useful.
-Eric
UpdateI have managed to get FatFS working and I can read my SD card. FatFS looks good. It already supports re-entrancy so should be OK in RTOS.
So now I have all the components working, I've just got to make them all play nicely together!
Working:
- uip including dhcpc, telnet, httpd (but not all at the same time)
- FatFS disk access (currently just listing the root dir)
- Simple serial output
TODO:
- Serial debug output in a separate task (I have a design in mind for this)
- Multiple simultaneous uip tcp tasks (I know how to do this)
- Fix stack overflows (I hope this is being caused by multiple threads accessing the serial port - not a good idea I know. Hence item 1 on the TODO list!)
- Investigate issue with packet size sent from httpd (Don't know the cause of this yet)
- Web server reading from sd card
[quote author="mattcallow"]
Update
I have managed to get FatFS working and I can read my SD card. FatFS looks good. It already supports re-entrancy so should be OK in RTOS.
So now I have all the components working, I've just got to make them all play nicely together!
...
[/quote]
Matt,
Great news. I placed my order for a Web Platform (two actually) so whenever I get them I will start testing/experimenting with this. I may eventually use some of this in my thesis project so I can tell myself this is school work so I don't feel like I shouldn't be spending too much time on it.
-Eric
[quote author="mattcallow"]
Serial debug output in a separate task (I have a design in mind for this)
[/quote]
I have basic logger working
[quote author="mattcallow"]
Web server reading from sd card
[/quote]
This is working, but needs more work to get the script parts working
The code is still a way off being stable, and ready for use. But it's getting there...
I'd like to check it out. Would you please check it into our SVN or git, or somewhere?
[quote author="ian"]
I'd like to check it out. Would you please check it into our SVN or git, or somewhere?
[/quote]
I can put in in svn. How do i get commit access?
The FreeRTOS port is now in svn: trunk/web-platform/FreeRTOS
It's by no means complete, but it does work.
enjoy
[quote author="mattcallow"]
The FreeRTOS port is now in svn: trunk/web-platform/FreeRTOS[/quote]
Good news. As soon as I get a Web Platform I'll start working with it.
-Eric
I just got my Web Platform this week. Now, as soon as I can find some time, I will start helping with the FreeRTOS implementation. Has there been any recent progress?
-Eric
I messed with it for an afternoon. I'd like to use it in our annual interactive web-art project, but I'm not sure what it will be yet :)
[quote author="Eric"]
I just got my Web Platform this week. Now, as soon as I can find some time, I will start helping with the FreeRTOS implementation. Has there been any recent progress?
-Eric
[/quote]
I've been working on the web server - to provide a script interface. I've not had much time to work on this, and it's not ready yet - but it's almost there.
How about eLua scripting support ?
http://www.eluaproject.net (http://www.eluaproject.net)
[quote author="robots"]
How about eLua scripting support ?
http://www.eluaproject.net (http://www.eluaproject.net)
[/quote]
I think that's unlikely.
I couldn't see a port for dsPICs. From the FAQ:
A stripped down, integer-only version of eLua can definitely fit in 128k of Flash and depending on your type of application, 32k of RAM might prove just fine. We have built eLua for targets with less than 10K RAM but you can't do much than blinking an LED with them. It really largely depends on your needs.
(The web platform has 128k of Flash and 8k of RAM)
I won't be trying to port it any time soon...
I was thinking more along the lines of allowing C functions to be called from web pages, much like the microchip stack. In fact the uIP web server already does this, but I'm not keen on the implementation. What I'm working on is allowing tags in the html like
<% fn args%>
When the web server sees this it calls fn(args).
The function fn() does whatever it likes, (including possibly reading/writing to FreeRTOS Qs) and then writes the result to the uIP buffer. When it's finished, the web server continues with the rest of the page.
I have started to implemented this, so I can control the LEDs on the board from a browser but it's currently a bit unstable.
I think some simple functions to read/write to specific memory locations will be enough for most purposes, but the general principal can be extended as required.
I had a little time last night to start working with this. All I have really done so far was get the code from SVN, get the project configured in MPLAB 8, get it to compile and program the PIC. I did then test that the serial console is spitting out log messages and the uIP stack is acquiring a DHCP assigned IP address but that was as far as I got.
I think I should have some time this weekend to work on it. Matt, I'll leave the web server part to you. I think I will look at trying to get the telnet server working and tied in with the serial console if I can. If there is anything else you would like me to take a look at let me know.
-Eric
Matt,
I have now started working my way through the code.
I was originally thinking I would start with the telnet server but I think there are a few other things I want to do first. The first thing is to get the logger and console working a little better. I see you started writing a logging task that will accept log messages via a queue; I will try to get that going. I want to integrate the hardware CRC peripheral instead of the software routine that uIP uses. I also want to look at the 32 bit math code of uIP and see if there is a better way to implement it as well.
In the only software collaboration I have ever done I was writing some functions that someone else would be using. There was very little interaction other than to agree on the function arguments and return values. I just wanted to say this so you know I don't really know how software collaboration is supposed to work.
The first thing I usually do when I start learning code someone else wrote and adapting it for my own use is to go through it and clean up all the formatting, clean out all the dead and left over code, add comments, remove unneeded variables, eliminate unneeded function nesting, fix all the compiler warnings, etc. Basically just simplify and clean up the code. I think this is now called refactoring but I could be wrong. I don't know if this would be appreciated or considered rude on collaborative project like this one. I don't mind cleaning things up because it helps me learn the code as I go through it but if it is considered bad form I will leave things alone.
My first questions are about the logger. Why did you design the log_message function to take a variable printf style argument list instead of just a char pointer? This seems a little overkill for this application but you may have had a good reason. I would have the log_message function take a char pointer and then copy the string to a queue. I think the application writing the log message should take care of any string formatting it wants and the logger should just handle printable strings. Did you start writing this logger from scratch or were you adapting something else? Let me know if you have any strong opinions on how this is done.
I am also starting to think about how to incorporate the logger with both the serial console and a telnet session. I would like to have both running the same shell code. In something this small I do not think there is a need to handle multiple shell sessions so I am thinking that the serial console and a telnet session will interact with one shell thread. They will both see the same output and one command interpreter will handle input from both at the same time. The character echoing and command buffers would have to be separate though since this will be local for the serial console and handled by the user's terminal for telnet. Let me know if you have any thoughts on this.
-Eric
[quote author="Eric"]The first thing I usually do when I start learning code someone else wrote and adapting it for my own use is to go through it and clean up all the formatting, clean out all the dead and left over code, add comments, remove unneeded variables, eliminate unneeded function nesting, fix all the compiler warnings, etc. Basically just simplify and clean up the code. I think this is now called refactoring but I could be wrong. I don't know if this would be appreciated or considered rude on collaborative project like this one. I don't mind cleaning things up because it helps me learn the code as I go through it but if it is considered bad form I will leave things alone.[/quote]There are many different levels, and so there is not one perfect answer to your question.
First of all, any change at all to working code is risky, because it can introduce bugs. Unless you are prepared to thoroughly test every single feature where you alter the source code, it's best to avoid making changes. I would suggest that if you are not improving the capabilities of a function, then you should not even change it.
Second of all, changing formatting from one accepted style to another is considered bad form in a collaborative project. But this assumes that the existing files are consistently formatted and that the original authors are highly attached to their original coding style. Theoretically, if you have a suggestion for improving the formatting style, and you can get agreement from the active participants, then you could solicit permission from the group to make your changes. However, I would suggest that this be done in one single massive submit, not slowly over several feature changes.
Finally, if there are actual problems in the existing source code, and you can make the formatting more consistent while honoring the preferences of most contributors, and especially if you can also improve the code by reviewing it professionally, then any open source project would welcome such an effort.
Code review can be one of the best things for a project, but only if done at an appropriate time, and only when you understand the potential ramifications.
I think I know how you feel, though, because I often cannot resist global search and replace for annoying things like trailing white space on lines, inconsistent indentation, random white space before and after parenthetical operators, et cetera. If you plan on submitting any of your changes, I suggest that you first write the group and list the sorts of things you plan on changing before making massive edits to the source. There's nothing worse than a huge source code revision diff which is weighted heavily towards formatting, but still has functional changes hidden among the noise.
Well, after trying to layout how I was going to implement the serial console and telnet server I started doing some reading. Like a fractal, this seemed simple from a distance but the more I look into the details the more complicated it gets -- the devil is in the details as they say. My assumption that there was one mode of character echoing that a telnet connection operates in was wrong. There are at least five permutations of local and remote echoing and either one or both end may or may not support or request echoing from the other end. I'm reading through the relevant RFCs right now because I could not find any other concise description of how the protocol is supposed to work. While several of these go back to 1971 it turns out telnet is even older but wasn't really documented till then. The sample telnet daemon in uIP does not support switching modes so I have to work out how to handle it. At this point I'm not even sure what mode it is using or what it is expecting the other end to use.
Since I am only planning to have one shell instance there will be a lot of coupling between the serial console and telnet server code so I want to make sure I have it worked out in my head how it is all going to work before I start writing any code. Maybe by tomorrow I can actually start coding something.
-Eric
[quote author="rsdio"]
There are many different levels, and so there is not one perfect answer to your question.[/quote]
That's kind of what I figured.
[quote author="rsdio"]
...this assumes that the existing files are consistently formatted...[/quote]
...which they aren't.
[quote author="rsdio"]
Theoretically, if you have a suggestion for improving the formatting style, and you can get agreement from the active participants, then you could solicit permission from the group to make your changes.[/quote]
I think Matt and I are the only ones working on this so I guess I just need to find out if he has any strong opinions.
I don't really have any strong preference. I have seen many different coding styles and they all have benefits and drawbacks. I can work with any standard as long as somebody picks one.
[quote author="rsdio"]
However, I would suggest that this be done in one single massive submit, not slowly over several feature changes.[/quote]
I can easily see doing one whole file at a time as I work on them but I don't know about trying to do the whole project all at once.
[quote author="rsdio"]
I think I know how you feel, though, because I often cannot resist global search and replace for annoying things like trailing white space on lines, inconsistent indentation, random white space before and after parenthetical operators, et cetera.[/quote]
Yep.
[quote author="rsdio"]
If you plan on submitting any of your changes, I suggest that you first write the group and list the sorts of things you plan on changing before making massive edits to the source.[/quote]
I think I am... :) Thanks for your comments.
-Eric
[quote author="Eric"]
Well, after trying to layout how I was going to implement the serial console and telnet server I started doing some reading. Like a fractal, this seemed simple from a distance but the more I look into the details the more complicated it gets -- the devil is in the details as they say. My assumption that there was one mode of character echoing that a telnet connection operates in was wrong. There are at least five permutations of local and remote echoing and either one or both end may or may not support or request echoing from the other end. I'm reading through the relevant RFCs right now because I could not find any other concise description of how the protocol is supposed to work. While several of these go back to 1971 it turns out telnet is even older but wasn't really documented till then. The sample telnet daemon in uIP does not support switching modes so I have to work out how to handle it. At this point I'm not even sure what mode it is using or what it is expecting the other end to use.
Since I am only planning to have one shell instance there will be a lot of coupling between the serial console and telnet server code so I want to make sure I have it worked out in my head how it is all going to work before I start writing any code. Maybe by tomorrow I can actually start coding something.
-Eric
[/quote]
You do know that there is a telnet server in the uIP code base, dont you? It's not brilliant but it does work.
Matt
[quote author="Eric"]
I was originally thinking I would start with the telnet server but I think there are a few other things I want to do first. The first thing is to get the logger and console working a little better. I see you started writing a logging task that will accept log messages via a queue; I will try to get that going. I want to integrate the hardware CRC peripheral instead of the software routine that uIP uses. I also want to look at the 32 bit math code of uIP and see if there is a better way to implement it as well.
[/quote]
Getting the existing telnet server working would be pretty easy. I haven't done that because a) I don't have a need for it at the moment b) the uIP implementation is not that good - it does not handle lots of output very well. For example, if you try to list a directory on the SD card, the last line get over-written with each previous line rather than scrolling.
Hardware CRC and 32bit math would be good.
In the only software collaboration I have ever done I was writing some functions that someone else would be using. There was very little interaction other than to agree on the function arguments and return values. I just wanted to say this so you know I don't really know how software collaboration is supposed to work.
The first thing I usually do when I start learning code someone else wrote and adapting it for my own use is to go through it and clean up all the formatting, clean out all the dead and left over code, add comments, remove unneeded variables, eliminate unneeded function nesting, fix all the compiler warnings, etc. Basically just simplify and clean up the code. I think this is now called refactoring but I could be wrong. I don't know if this would be appreciated or considered rude on collaborative project like this one. I don't mind cleaning things up because it helps me learn the code as I go through it but if it is considered bad form I will leave things alone.
My comments on this:
- Be careful when changing code which uses proto-threads/proto-sockets. I tried to reduce the number of function calls in the web server - with bad results! You have to consider what the proto* macros are doing to manage the 'threads'
- Try to separate functional changes from refactory when commiting to svn. This makes it easier to see when new functionallity has been added, and easier to roll back a change if, for example, the refactoring breaks something
- Since this code is built up from lots of different components, there will always be different code standards/formats. I suggest that we try to keep the format the same as the existing files. e.g. for uIP stuff, use the uIP code style. For FreeRTOS stuff, use the FreeRTOS code style, for FatFS stuff, use the FatFS code style. Having said that, I acknowldge that I probably didn't follow that rule in my code.
My first questions are about the logger. Why did you design the log_message function to take a variable printf style argument list instead of just a char pointer? This seems a little overkill for this application but you may have had a good reason. I would have the log_message function take a char pointer and then copy the string to a queue. I think the application writing the log message should take care of any string formatting it wants and the logger should just handle printable strings. Did you start writing this logger from scratch or were you adapting something else? Let me know if you have any strong opinions on how this is done.
This logger is my design/fault. The reason for using printf style function is to reduce memory. The reasoning is as follows:
The logger runs in it's own thread. Therefore, it must copy have a local copy of all variables passed to it that it will use. Since they may go out of scope in the original thread. For this reason, the logger stores log messages on queue. If we store the whole message as a string, then we use up lots of RAM. My idea is to only store copies of the variables. The format string is a constant, so doesn't need to be copied.
For example, consider logging the following:
"Sent %x bytes on port %dn", 40, 80
If we use char* for the logger, we would need to copy 25 bytes for the message.
Using const char* and vargs, we only need to copy a pointer to the string (2 bytes) plus 4 bytes for the arguments (assuming they are shorts)
So we save 19 bytes of RAM per message.
I am also starting to think about how to incorporate the logger with both the serial console and a telnet session. I would like to have both running the same shell code. In something this small I do not think there is a need to handle multiple shell sessions so I am thinking that the serial console and a telnet session will interact with one shell thread. They will both see the same output and one command interpreter will handle input from both at the same time. The character echoing and command buffers would have to be separate though since this will be local for the serial console and handled by the user's terminal for telnet. Let me know if you have any thoughts on this.
I don't really understand why you would want to log to both the serial port and telnet?
[quote author="mattcallow"]
You do know that there is a telnet server in the uIP code base, dont you? It's not brilliant but it does work.[/quote]
Yes, I read through the code to try to understand what it was doing but have not attempted to compile it or run it yet. It seems to be so basic that it is more of a capability demonstration than a functional telnet server but I could be wrong. I have to study the uIP data structures and API a bit more to fully grasp what all it is doing or not doing.
[quote author="mattcallow"]
...the existing telnet server ...the uIP implementation is not that good - it does not handle lots of output very well.[/quote]
That was my impression also which is what led me down the rabbit hole of trying to understand the telnet protocol -- but I have learned a lot.
[quote author="mattcallow"]
Hardware CRC and 32bit math would be good.[/quote]
These should not be too hard but I would like to get the console working.
[quote author="mattcallow"]
My comments on this:
- Be careful when changing code which uses proto-threads/proto-sockets.[/quote]
I'll steer clear of the uIP code other than what I need to change in the telnet server. Overall it and the FatFs code looked pretty good. The FreeRTOS code is messy because they write it to support so many different processors. In my own projects I have done quite a bit of simplification of this code.
[quote author="mattcallow"]
- Try to separate functional changes from refactory when commiting to svn.[/quote]
Sounds good.
[quote author="mattcallow"]
- Since this code is built up from lots of different components... I suggest that we try to keep the format the same as the existing files.[/quote]
Also sounds good.
[quote author="mattcallow"]
This logger is my design/fault. The reason for using printf style function is to reduce memory. The reasoning is as follows:
The logger runs in it's own thread. Therefore, it must copy have a local copy of all variables passed to it that it will use. Since they may go out of scope in the original thread. For this reason, the logger stores log messages on queue. If we store the whole message as a string, then we use up lots of RAM. My idea is to only store copies of the variables. The format string is a constant, so doesn't need to be copied.
For example, consider logging the following:
"Sent %x bytes on port %dn", 40, 80
If we use char* for the logger, we would need to copy 25 bytes for the message.
Using const char* and vargs, we only need to copy a pointer to the string (2 bytes) plus 4 bytes for the arguments (assuming they are shorts)
So we save 19 bytes of RAM per message.[/quote]
This method is only beneficial if the const format string is a significant portion of the output. The uIP logging function, uip_log, for example does not take advantage of this so, as you have had to do in uip_task.c, it will always be "%s", m. My guess is that most any other code that we use or convert will be expecting to spit its messages directly out to some serial port and so will not be able to easily take advantage of this without us having to rewrite each call instead of just one function but I could be wrong. I just don't think the added complexity will be worth the little memory savings we will see in actual use. Also, I have not studied what you have so far but each slot in the queue would have to be big enough to hold the longest string. This is a fixed length so there is no memory savings once the queue is created. I guess since the logger will be writing to the console asynchronously anyways both methods could easily coexist.
[quote author="mattcallow"]
I don't really understand why you would want to log to both the serial port and telnet?[/quote]
Because if the telnet connection is the interface I am using to try to troubleshoot some higher level function or application I would like to be able to see the log/error messages that I would see if I was connected via serial console. And since the log messages will not be written to a file -- that is probably overkill -- I think the only easy way to get them is to (possibly optionaly) have them print to the telnet connection as well. The only limitation of the telnet connection is that it would not be available early in the "boot" process. That and if you crashed the network you might still have the serial console.
Speaking of crashing the network, I have found that many/most times if I request a nonexistent page the PIC will reset. Sometimes I get my 404 page but most times it hangs.
Web Platform
Last reset was 0080 external
SD Power on
CardType=4
0 disk_initialize(0) returns 0
f_mount(0) returns 0
Reading SD Card...
SD Power on
CardType=4
fs_type=2 csize=64
NIC DMA Init
nic init
uip init
arp init
dhcp configured. My IP=10.0.0.134
opening /INDEX.HTM
Calling generator fn
Copied 229 bytes. Tried to copy 229 bytes
Waiting for data to be sent. sendlen=229
Data sent. sendlen=0
Calling generator fn
Copied 81 bytes. Tried to copy 81 bytes
Waiting for data to be sent. sendlen=81
Data sent. sendlen=0
opening /a
opening /404.HTM
Calling generator fn
Copied 211 bytes. Tried to copy 211 bytes
Waiting for data to be sent. sendlen=211
Web Platform
Last reset was 0000 SD Power on
In the case above I rest the PIC, requested 10.0.0.134/ and got the index.htm file as expected. I then requested 10.0.0.134/a and it tried to send the 404.htm file but it crashes and resets. The browser gets part of the 404 page because it changes the browser title but never displays the page -- the wheel just keeps spinning. I have not tried to look in to the cause of this.
-Eric
[quote author="Eric"]
[quote author="rsdio"]However, I would suggest that this be done in one single massive submit, not slowly over several feature changes.[/quote]
I can easily see doing one whole file at a time as I work on them but I don't know about trying to do the whole project all at once.[/quote]On second thought, I really just suggest that you keep formatting changes separate from other changes. It's not really necessary to get all formatting changes into one submit for the whole project, but at least try not to submit multiple formatting change revisions to a single file. These are just my general opinions - I really have nothing to do with this project.
Matt has already suggested this, with the addition that refactoring should also be separate from other changes. That's great advice.
Just offering my thoughts,
Is it really a Real-Time (preemptive) OS you need/want for the web platform?
How about contiki (www.sics.se/contiki (http://http://www.sics.se/contiki))? It is Adam Dunkels continuation of protothreads/uIP taken to a whole new level as a cooperative multithreading operating system.
Contiki is geard towards wireless sensor networks and there is a whole lot of IPv6 PAN routing protocool development beeing conducted in it right now, but it is supposed to be somewhat "pure" C99 and I've seen multiple ports beeing developed in a short amount of time. There were even a weekend class of porting it to your favorite platform a couple of years ago.
I don't want to steer you away from RTOS, but offer an possibly easier alternative if all you want is some process abstractions.
[quote author="honken"]
How about contiki (www.sics.se/contiki (http://http://www.sics.se/contiki))? [/quote]
I just took a very quick look at it but it seems like it is quite a bit heavier in terms of both code space and memory requirements than FreeRTOS + uIP but I could be wrong.
In my case I have used FreeRTOS before and so am somewhat familiar with it. Also, when you do not need network connectivity FreeRTOS is tiny, only three source files I if I remember correctly.
-Eric
[quote author="honken"]
Just offering my thoughts,
Is it really a Real-Time (preemptive) OS you need/want for the web platform?
How about contiki (www.sics.se/contiki (http://http://www.sics.se/contiki))? It is Adam Dunkels continuation of protothreads/uIP taken to a whole new level as a cooperative multithreading operating system.
Contiki is geard towards wireless sensor networks and there is a whole lot of IPv6 PAN routing protocool development beeing conducted in it right now, but it is supposed to be somewhat "pure" C99 and I've seen multiple ports beeing developed in a short amount of time. There were even a weekend class of porting it to your favorite platform a couple of years ago.
I don't want to steer you away from RTOS, but offer an possibly easier alternative if all you want is some process abstractions.
[/quote]
I did look at this - briefly. The big advantage of FreeRTOS for me was the existing dsPIC port. I couldn't find any such port for contiki.
Matt
[quote author="Eric"]
This method is only beneficial if the const format string is a significant portion of the output. The uIP logging function, uip_log, for example does not take advantage of this so, as you have had to do in uip_task.c, it will always be "%s", m. My guess is that most any other code that we use or convert will be expecting to spit its messages directly out to some serial port and so will not be able to easily take advantage of this without us having to rewrite each call instead of just one function but I could be wrong.
[/quote]
The uip_log was my quick and dirty implementation of a function that uip requires in each port. We could easily replace all these calls with direct call to the logger.
[quote author="Eric"]
I just don't think the added complexity will be worth the little memory savings we will see in actual use. Also, I have not studied what you have so far but each slot in the queue would have to be big enough to hold the longest string. This is a fixed length so there is no memory savings once the queue is created. I guess since the logger will be writing to the console asynchronously anyways both methods could easily coexist.
[/quote]
In the end, the added complexity may make it impractical, but memory is quite tight at the moment...
[quote author="Eric"]
Speaking of crashing the network, I have found that many/most times if I request a nonexistent page the PIC will reset. Sometimes I get my 404 page but most times it hangs.
Web Platform
Last reset was 0080 external
SD Power on
CardType=4
0 disk_initialize(0) returns 0
f_mount(0) returns 0
Reading SD Card...
SD Power on
CardType=4
fs_type=2 csize=64
NIC DMA Init
nic init
uip init
arp init
dhcp configured. My IP=10.0.0.134
opening /INDEX.HTM
Calling generator fn
Copied 229 bytes. Tried to copy 229 bytes
Waiting for data to be sent. sendlen=229
Data sent. sendlen=0
Calling generator fn
Copied 81 bytes. Tried to copy 81 bytes
Waiting for data to be sent. sendlen=81
Data sent. sendlen=0
opening /a
opening /404.HTM
Calling generator fn
Copied 211 bytes. Tried to copy 211 bytes
Waiting for data to be sent. sendlen=211
Web Platform
Last reset was 0000 SD Power on
In the case above I rest the PIC, requested 10.0.0.134/ and got the index.htm file as expected. I then requested 10.0.0.134/a and it tried to send the 404.htm file but it crashes and resets. The browser gets part of the 404 page because it changes the browser title but never displays the page -- the wheel just keeps spinning. I have not tried to look in to the cause of this.
-Eric
[/quote]
Yeah, i get that sometimes. But not always. Like I said, it's not that stable yet.
The reason it hangs completely after the reset is due to the SD card init. I think this needs to be re-done. My idea if to let any use a mutex in the init call, so that any thread can call it. Currently, it's called before the scheduler is started.
[quote author="mattcallow"]
...but memory is quite tight at the moment...[/quote]
This was one of the things that struck me as strange the first time I compiled the project but I have not looked in to it too far yet. The uIP code seems to have a lot of (and many big) variables but I imagine this has already been scrutinized quite a bit by others already. The other big target is the heap. Do we need 3kB of heap space? I have not looked yet but what requires this much dynamic memory allocation?
...
Okay, I just took a quick look and you are using heap_3.c in FreeRTOS. I have always used heap_1.c before in my projects. What drove the need for true dynamic memory? Was it the uIP stack or the FatFs file system?
...
Okay, another quick look says they are both using it. I do not think this is something I will get too soon but but my guess is that they do not need that much. The amount needed by the file system should be very predictable based on how many files you need to have open at once (1 or 2?) and whether or not it is doing any buffering. The uIP code I have no idea about what it should actually need.
-Eric
How is this effort proceeding? Is more help required? I may have some time in a few weeks to help if it makes sense.
[quote author="rhyde"]
How is this effort proceeding?
[/quote]
Well...
I guess I should start by saying I failed miserably...
...that is I failed miserably at not making major functional changes and wholesale format changes at the same time.
The more I went through the code the more I found things that weren't quite right or were just ugly. (edit: I should have added "or were things that I just didn't like.") Trying to hack fixes in would only have made the situation worse. So I decided to forget about trying not to change things too much. I also used a bunch of code I wrote for other projects that does initialization and what not.
So other than completely rearranging the source files and directory structure I have:
- Fixed the oscillator PLL switching. The original code was not unlocking the OSCCON low and high registers before writing them so the PLL was never changed from the reset default. The defaults would have it running at over 92MHz (80MHz Max) so I don't know if it was actually running overclocked or if it would somehow end up running slower since parts of the oscillator circuit may not have been able to keep up. I added a couple files (osc.c/h) I use for configuring the oscillator to make it cleaner.
- The configuration bits were being set in two different files and were not quite correct. For example, the OSC2/RA3 pin was not set as a digital pin as is needed by the new v1.1 hardware. There may have been something other issue too -- I don't remember now. Again, I added a file (configbits.h) I use to keep this clean.
- I cleaned up the peripheral pin select setup and moved it to separate files (pps.c/h & ppsdef.h) that use standard macros to make the configuration a little easier and more obvious.
- I implemented the check for whether a USB host is attached and the FTDI chip is powered before enabling the serial port transmitter. There was a note about this in the source but it was not being done. Right now it only checks one time on start up and turns it on or off permanently. I will try to make this a dynamic check later.
- The value of BRG was one count low. It was 0.9% high before now it is 0.2% low. Though either of these are well within spec.
- I wrote a temporary replacement for the standard library write() function because the Microchip included version would always enable the UART transmitter. If I have it off because there is no USB host attached I do not want a call to printf() to turn it on. Eventually this will interface with the system logger. It seemed cleaner and easier to make the change once in write() instead of changing each call to printf() for now. Once I get to work on the logger I may want to do this differently but lots of code seems to just use a printf() to spit out debugging info so this might end up being the best solution anyway.
- I added interrupt driven serial code with circular transmit and receive buffers. This took a little longer than it should have because the routines I had been using in my projects were copied out of a book and so I thought I should rewrite them from scratch for this. Every time I try to code a circular buffer I end up confusing myself and so have to work through the code with paper and a pencil to make sure all the index pointer math is correct. I have versions of putc() and getc() implemented so far.
This is where I am at as of right this minute. I am just now starting to write the console/terminal code to create the user interface. Once I get this working with the serial port I will also connect it to the telnet server.
[quote author="rhyde"]
Is more help required?
[/quote]
Right now I don't know what is going to happen with the code I am working on. Since Matt started this project I guess it is kind of his call what he does with it. I am working on this because I know I will use it myself. I will post it here once I get to a good stopping point and then people can look at it. It will not be able to be merged back into SVN because it is too different. Whether Matt wants to port any of my work to the SVN version is up to him. If anyone else actually wants to use it I guess we could call it a fork and see if Ian wants to add it to SVN. If Matt really wanted to I suppose he could use my version. Otherwise it will just be a personal project.
[quote author="rhyde"]
I may have some time in a few weeks to help if it makes sense.
[/quote]
By that point I should have a version or two of mine posted here. I don't know how much time Matt has to devote to this. I know he wants to get the stability of the web server fixed. That might be a good area to look at. Any work done on the uIP code would be useful to everyone. I don't expect to be mucking around in the uIP code if I can avoid it. With the possible exception of trying to figure out how much dynamic memory it really needs and reducing the heap.
-Eric
[quote author="Eric"]
Right now I don't know what is going to happen with the code I am working on. Since Matt started this project I guess it is kind of his call what he does with it. I am working on this because I know I will use it myself. I will post it here once I get to a good stopping point and then people can look at it. It will not be able to be merged back into SVN because it is too different. Whether Matt wants to port any of my work to the SVN version is up to him. If anyone else actually wants to use it I guess we could call it a fork and see if Ian wants to add it to SVN. If Matt really wanted to I suppose he could use my version. Otherwise it will just be a personal project.
[/quote]
All the code is open source, so people can change it as they wish. It seems a shame to create a fork of the existing code, but so be it.
[quote author="Eric"]
By that point I should have a version or two of mine posted here. I don't know how much time Matt has to devote to this. I know he wants to get the stability of the web server fixed. That might be a good area to look at. Any work done on the uIP code would be useful to everyone. I don't expect to be mucking around in the uIP code if I can avoid it. With the possible exception of trying to figure out how much dynamic memory it really needs and reducing the heap.
[/quote]
I think it may be possible to reduce the heap size. I don't know much is really needed. The reason for using heap_3.c rather than either of the other heap models is an effort to consolidate the dynamic memory allocation schemes in use, which (i think) will lead to better use of the available memory. Currently, the microchip libraries (e.g. printf) will use dynamic memory (i assume via malloc/free), uIP has it's own memory blocks (used in the telnet server), FreeRTOS needs to use one of heap_1/2/3.c and FatFS will need dynamic memory for LFN. Using the heap as the underlying dynamic memory pool for all these schemes seemed like the best way to go (in terms of memory usage).
Matt
[quote author="mattcallow"]
It seems a shame to create a fork of the existing code, but so be it. [/quote]
Agreed. I do not necessarily want to create a fork (I just threw it out as an option) which is why I am just going to call this a personal project and offer anything I write back to the forum. I guess all I am really saying is that I do not want to try to back-port (or whatever it's called) my changes in to the SVN version because my code is now so different. If you want to merge any of my stuff in great, if not don't worry about it. If you like what I end up with enough you could possibly even switch the code in SVN to it if you felt so inclined. I have never used SVN other than for just downloading code so I don't know how it would deal with the large structural changes I've made.
[quote author="mattcallow"]
I think it may be possible to reduce the heap size. I don't know much is really needed. The reason for using heap_3.c rather than either of the other heap models is an effort to consolidate the dynamic memory allocation schemes in use, which (i think) will lead to better use of the available memory. Currently, the microchip libraries (e.g. printf) will use dynamic memory (i assume via malloc/free), uIP has it's own memory blocks (used in the telnet server), FreeRTOS needs to use one of heap_1/2/3.c and FatFS will need dynamic memory for LFN. Using the heap as the underlying dynamic memory pool for all these schemes seemed like the best way to go (in terms of memory usage).[/quote]
In the last couple hours I started bumping up against the the memory limit so I began investigating the issue. It turns out uIP does not use any dynamic memory (as you noted above). It statically declares all the buffers it needs at compile time. FatFs also does not use dynamic memory unless long file name (LFN) support is turned (as you also noted). Even with LFN support enabled it looks like it has an option to just use the stack. So FatFs my never need dynamic memory. If it does you need to write its ff_malloc() for it anyway. I switched FreeRTOS to heap_1.c (which is what I have always used) since FreeRTOS never frees memory unless you delete a task or a queue -- and I can't think of a case where you would ever want to do this. This let me set the compiler heap to zero. You can't disable it or the linker will complain if you use any stdio.h input or output functions. But if you never need a FILE * pointer (like if you only use printf() to a serial port) you can set the heap to zero.
I also noticed that the the configMINIMAL_STACK_SIZE was set to 256 bytes and the uIP task was using twice this value. This is probably twice what most tasks need but it turns out it was a good value for the uIP task. Was this a guess or did you measure it? I set the configMINIMAL_STACK_SIZE back down to 115, the default for the dsPIC/PIC24 port, but created a configUIP_STACK_SIZE of 512 for the uIP task. I measured the uIP stack grow to a max of 498 so 512 is probably a good value for now. We'll need to keep an eye on this and see if it gets any closer to the top of the stack. With the configTOTAL_HEAP_SIZE set at 2048 and the uIP task and two other tasks (the idle task and my console task) of default stack size there are still 310 bytes left in the FreeRTOS heap -- probably enough for two more normal sized tasks. Of course the configTOTAL_HEAP_SIZE could just be brought down to make this go to zero if desired since once all tasks and all queues are created no more memory is ever needed.
All this brought the total memory usage down to about 79% from what ever it was before -- ninety-something I think.
Now that I have some breathing room I'll get back to what I was working on.
-Eric
Eric,
Sounds like a ton of progress. I can't help but make code my own either, especially if I want to understand it thoroughly.
Let's get you SVN access so you can keep your progress in a separate folder or something. I'm sure there's other people (me :) ) who would like to play along at home
Here's the info about getting a password for Google Code:
http://dangerousprototypes.com/docs/Using_SVN (http://dangerousprototypes.com/docs/Using_SVN)
[quote author="Eric"]
In the last couple hours I started bumping up against the the memory limit so I began investigating the issue. It turns out uIP does not use any dynamic memory (as you noted above). It statically declares all the buffers it needs at compile time. FatFs also does not use dynamic memory unless long file name (LFN) support is turned (as you also noted). Even with LFN support enabled it looks like it has an option to just use the stack. So FatFs my never need dynamic memory. If it does you need to write its ff_malloc() for it anyway. I switched FreeRTOS to heap_1.c (which is what I have always used) since FreeRTOS never frees memory unless you delete a task or a queue -- and I can't think of a case where you would ever want to do this. This let me set the compiler heap to zero. You can't disable it or the linker will complain if you use any stdio.h input or output functions. But if you never need a FILE * pointer (like if you only use printf() to a serial port) you can set the heap to zero.
[/quote]
Good to know that printf doesn't need the heap for serial port output. I'm still not sure about using the stack for LFN, since that would require a large stack for each task using FatFS. Anyway, currently it doesn't matter since LFN is disabled.
[quote author="Eric"]
I also noticed that the the configMINIMAL_STACK_SIZE was set to 256 bytes and the uIP task was using twice this value. This is probably twice what most tasks need but it turns out it was a good value for the uIP task. Was this a guess or did you measure it? I set the configMINIMAL_STACK_SIZE back down to 115, the default for the dsPIC/PIC24 port, but created a configUIP_STACK_SIZE of 512 for the uIP task. I measured the uIP stack grow to a max of 498 so 512 is probably a good value for now. We'll need to keep an eye on this and see if it gets any closer to the top of the stack. With the configTOTAL_HEAP_SIZE set at 2048 and the uIP task and two other tasks (the idle task and my console task) of default stack size there are still 310 bytes left in the FreeRTOS heap -- probably enough for two more normal sized tasks. Of course the configTOTAL_HEAP_SIZE could just be brought down to make this go to zero if desired since once all tasks and all queues are created no more memory is ever needed.
[/quote]
Stack size was just a guess. I was, at one point, going to put the uIP code in the idle task. In that case, i needed to increase configMINIMAL_STACK_SIZE, because that's the only way (i know) of changing the stack size of the idle task. Since uIP is running in it's own task, the change is not required.
Maybe once you've posted your code, I'll try some of this to see if it makes the stack more stable.
Matt
[quote author="ian"]
Sounds like a ton of progress.[/quote]
Thanks. I finally got in the programming "groove" and have started to move forward now.
[quote author="ian"]
I can't help but make code my own either, especially if I want to understand it thoroughly. [/quote]
Thanks for understanding. It was certainly never my intent to create division and I owe Matt a debt of gratitude for getting this thing kicked off. I don't want all the changes I have made to come across in any way negative. Matt did the real hard work here of getting this thing going. He got all the pieces together and functional which is where I tend to get bogged down. Now that it is running it is easy for me to make incremental improvements which I am much better at.
[quote author="ian"]
Let's get you SVN access so you can keep your progress in a separate folder or something. ... Here's the info...[/quote]
I'll read up on it and try to learn how to use SVN.
[quote author="ian"]
I'm sure there's other people (me :) ) who would like to play along at home[/quote]
I'm not quite there yet. There are still some things I want to go through first because they will probably cause major structural changes. Like I just noticed that the SD code is setting up Time 1 but Timer 1 is then later configured by the FreeRTOS scheduler when it starts. So right now there are two different initializations for Timer 1 in two different sections of code. I need to go through this and figure out what FatFs needs. I will probably move them on to different timers. I have done this before with FreeRTOS, moved it to Timer 5 because Timer 1 is used by certain hardware peripherals like the ADC. Then the scheduler "tick" can be decoupled from the file system timing and appropriate intervals can be set for each independently.
So till I get through a few more things like this there is still likely to be large code changes.
I also need to figure out what to do about another licensing issue. My uart.c file has a lot of code in it from the Microchip peripheral libraries that come with the C30 compiler. Some time ago I took the relevant library functions, which were each in their own file, and consolidated them in to one file with some code I added. The library routines were written for the most general case (like the ability to deal with 9 bit data) that I knew I would never need so I cleaned them up a bit. Since everyone with the C30 compiler already has the peripheral libraries I can probably just pull the library functions back out of my file. The only downside is that you are back to the bulky generic versions -- that and I like to have ALL the relevant source in the project directory where I can see and touch it -- it also really bugs me that Microchip does not include the source for the C standard library they include. The other option is for me to rewrite the file from scratch. I have always just written code for myself before and so never worried about copyright or licensing issues. Any thoughts on this?
[quote author="mattcallow"]
I'm still not sure about using the stack for LFN, since that would require a large stack for each task using FatFS. [/quote]
I'm not really sure yet either. We'll have to figure that out when we get to it. Since we have to write the ff_malloc() and free commands anyway we have a lot of flexibility.
[quote author="mattcallow"]
Stack size was just a guess. [/quote]
Good guess then.
[quote author="mattcallow"]
Maybe once you've posted your code, I'll try some of this to see if it makes the stack more stable.[/quote]
That would be great.
I also found there was a project setting somewhere with a flag something like _ICD2RAM = 1 or similar. Removing that also freed up some more memory. I don't know if you have an ICD2 or if that was left over from somewhere else. I think this flag should normally be enabled and disabled by MPLAB as you turn debugging on and off. I have an ICD3 and not an ICD2 so my MPLAB may not have known what that flag was for and assumed it was a user flag.
-Eric
[quote author="Eric"]
...Like I just noticed that the SD code is setting up Time 1 but Timer 1 is then later configured by the FreeRTOS scheduler when it starts. So right now there are two different initializations for Timer 1 in two different sections of code. I need to go through this and figure out what FatFs needs. I will probably move them on to different timers. I have done this before with FreeRTOS, moved it to Timer 5 because Timer 1 is used by certain hardware peripherals like the ADC. Then the scheduler "tick" can be decoupled from the file system timing and appropriate intervals can be set for each independently.
[/quote]
This is because I initialise the SD card before the scheduler is started. Because the code to init the SD card is not thread safe.
This is really a hack. As I mentioned earlier, probably the best way is to make the SD init code thread safe, add a 'initialised' variable, and allow it to be called from any thread.
[quote author="Eric"]
I also found there was a project setting somewhere with a flag something like _ICD2RAM = 1 or similar. Removing that also freed up some more memory. I don't know if you have an ICD2 or if that was left over from somewhere else. I think this flag should normally be enabled and disabled by MPLAB as you turn debugging on and off. I have an ICD3 and not an ICD2 so my MPLAB may not have known what that flag was for and assumed it was a user flag.
[/quote]
I only have pickit2. I expect that is a hangover from the original FreeRTOS port.
Matt
[quote author="mattcallow"]
[quote author="Eric"]
...I just noticed that the SD code is setting up Time 1 but Timer 1 is then later configured by the FreeRTOS scheduler...[/quote]
This is because I initialise the SD card before the scheduler is started. Because the code to init the SD card is not thread safe.
This is really a hack. As I mentioned earlier, probably the best way is to make the SD init code thread safe, add a 'initialised' variable, and allow it to be called from any thread. [/quote]
Good to know, thanks. I'll keep that in mind.
[quote author="mattcallow"]
[quote author="Eric"]
I also found there was a project setting somewhere with a flag something like _ICD2RAM = 1 or similar. Removing that also freed up some more memory.[/quote]
I only have pickit2. I expect that is a hangover from the original FreeRTOS port.[/quote]
It wasn't much but every little bit helps -- especially when it was just being wasted.
-Eric
[quote author="Eric"]... I just noticed that the SD code is setting up Time 1 but Timer 1 is then later configured by the FreeRTOS scheduler when it starts. So right now there are two different initializations for Timer 1 in two different sections of code. I need to go through this and figure out what FatFs needs. I will probably move them on to different timers. I have done this before with FreeRTOS, moved it to Timer 5 because Timer 1 is used by certain hardware peripherals like the ADC. Then the scheduler "tick" can be decoupled from the file system timing and appropriate intervals can be set for each independently.
So till I get through a few more things like this there is still likely to be large code changes.[/quote]Matt may have already answered this for the specific case at hand, but I wanted to mention a general technique. First of all, the PIC usually has a lot of timers, so if you can use Timer 5 and get away with it, then you're done. But sometimes you still end up running out of timers. What I do in that case is find a faster timer which is a reasonable multiple of another timer, and then create a virtual timer. Just use a volatile global variable to count the faster timer interrupts and then call the slower timer function as if a virtual timer had fired. The scheduler "tick" is probably not something which has to run at a precise speed - unlike PWM or a serial port - and so the scheduler tick is a good candidate for a virtual timer. You might already know these tricks, but I thought I'd pass them on anyway.
[quote author="rsdio"]
First of all, the PIC usually has a lot of timers, so if you can use Timer 5 and get away with it, then you're done. But sometimes you still end up running out of timers. What I do in that case is find a faster timer which is a reasonable multiple of another timer, and then create a virtual timer. Just use a volatile global variable to count the faster timer interrupts and then call the slower timer function as if a virtual timer had fired. The scheduler "tick" is probably not something which has to run at a precise speed - unlike PWM or a serial port - and so the scheduler tick is a good candidate for a virtual timer. You might already know these tricks, but I thought I'd pass them on anyway.
[/quote]
I have never run out of timers before but I'll keep this in mind. Usually it is just an issue of a particular peripheral needing to use a specific timer that necessitates moving other things around.
-Eric
[quote author="Eric"]
I also need to figure out what to do about another licensing issue. My uart.c file has a lot of code in it from the Microchip peripheral libraries that come with the C30 compiler.
[/quote]
I rewrote all my serial code now from scratch so there is no more Microchip peripheral library source code in it.
This gets me one step closer to being able to commit this to SVN. Hopefully this weekend I will have it ready.
-Eric
hi :)
noob with microchip PIC but i did few realtime systems long time ago...
so im trying first to compiling FreeRTOS from the trunk and i got linker error
Executing: "C:Program FilesMicrochipmplabc30v3.25binpic30-gcc.exe" -mcpu=33FJ128GP204 "main.o" "rtcc.o" "enc28j60.o" "nic_dma.o" "delay.o" "nic.o" "uip_task.o" "ff.o" "heap_3.o" "croutine.o" "list.o" "uip.o" "uip_arp.o" "timer.o" "portasm_dsPIC.o" "mmc.o" "memb.o" "port.o" "queue.o" "psock.o" "tasks.o" "httpd.o" "httpd-cgi.o" "http-strings.o" "dhcpc.o" "logger.o" -o"WebPlatform.cof" -Wl,--script="p33FJ128GP204.gld",--defsym=__MPLAB_BUILD=1,--defsym=__MPLAB_DEBUG=1,--heap=3096,--stack=512,--defsym=__ICD2RAM=1,-Map="WebPlatform.map"
heap: Link Error: Could not allocate section .heap, size = 3096 bytes, attributes = heap
Link Error: Could not allocate data memory
so ... someone has a hint for me ? :)
[quote author="voidptr"]
hi :)
noob with microchip PIC but i did few realtime systems long time ago...
so im trying first to compiling FreeRTOS from the trunk and i got linker error
Executing: "C:Program FilesMicrochipmplabc30v3.25binpic30-gcc.exe" -mcpu=33FJ128GP204 "main.o" "rtcc.o" "enc28j60.o" "nic_dma.o" "delay.o" "nic.o" "uip_task.o" "ff.o" "heap_3.o" "croutine.o" "list.o" "uip.o" "uip_arp.o" "timer.o" "portasm_dsPIC.o" "mmc.o" "memb.o" "port.o" "queue.o" "psock.o" "tasks.o" "httpd.o" "httpd-cgi.o" "http-strings.o" "dhcpc.o" "logger.o" -o"WebPlatform.cof" -Wl,--script="p33FJ128GP204.gld",--defsym=__MPLAB_BUILD=1,--defsym=__MPLAB_DEBUG=1,--heap=3096,--stack=512,--defsym=__ICD2RAM=1,-Map="WebPlatform.map"
heap: Link Error: Could not allocate section .heap, size = 3096 bytes, attributes = heap
Link Error: Could not allocate data memory
so ... someone has a hint for me ? :)
[/quote]
The memory usage is not good at the moment. Try reducing the heap size a little (somewhere on the linker tab in the IDE I think)
Eric has also pointed out that you can remove the __ICD2RAM=1 symbol
Matt
Is it ok to move your work to /port-matt/ in /FreeRTOS/? Then we can move Eric's work to /port-eric/, and create folders for anyone else's experiments too.
[quote author="ian"]
Is it ok to move your work to /port-matt/ in /FreeRTOS/? Then we can move Eric's work to /port-eric/, and create folders for anyone else's experiments too.
[/quote]
That's fine by me. Do you want me to move it, or will you?
Matt
It would be great if you can move it, but I can take care of it too :)
[quote author="ian"]
It would be great if you can move it, but I can take care of it too :)
[/quote]
done!
Thanks!
I also started a general wiki for the RTOS ports here:
http://dangerousprototypes.com/docs/Fre ... b_platform (http://dangerousprototypes.com/docs/FreeRTOS_on_the_web_platform)