[quote author="rsdio"] First of all, the PIC usually has a lot of timers, so if you can use Timer 5 and get away with it, then you're done. But sometimes you still end up running out of timers. What I do in that case is find a faster timer which is a reasonable multiple of another timer, and then create a virtual timer. Just use a volatile global variable to count the faster timer interrupts and then call the slower timer function as if a virtual timer had fired. The scheduler "tick" is probably not something which has to run at a precise speed - unlike PWM or a serial port - and so the scheduler tick is a good candidate for a virtual timer. You might already know these tricks, but I thought I'd pass them on anyway. [/quote]
I have never run out of timers before but I'll keep this in mind. Usually it is just an issue of a particular peripheral needing to use a specific timer that necessitates moving other things around.
[quote author="mattcallow"] [quote author="Eric"] ...I just noticed that the SD code is setting up Time 1 but Timer 1 is then later configured by the FreeRTOS scheduler...[/quote] This is because I initialise the SD card before the scheduler is started. Because the code to init the SD card is not thread safe. This is really a hack. As I mentioned earlier, probably the best way is to make the SD init code thread safe, add a 'initialised' variable, and allow it to be called from any thread. [/quote]
Good to know, thanks. I'll keep that in mind.
[quote author="mattcallow"] [quote author="Eric"] I also found there was a project setting somewhere with a flag something like _ICD2RAM = 1 or similar. Removing that also freed up some more memory.[/quote] I only have pickit2. I expect that is a hangover from the original FreeRTOS port.[/quote]
It wasn't much but every little bit helps -- especially when it was just being wasted.
[quote author="ian"] Sounds like a ton of progress.[/quote]
Thanks. I finally got in the programming "groove" and have started to move forward now.
[quote author="ian"] I can't help but make code my own either, especially if I want to understand it thoroughly. [/quote]
Thanks for understanding. It was certainly never my intent to create division and I owe Matt a debt of gratitude for getting this thing kicked off. I don't want all the changes I have made to come across in any way negative. Matt did the real hard work here of getting this thing going. He got all the pieces together and functional which is where I tend to get bogged down. Now that it is running it is easy for me to make incremental improvements which I am much better at.
[quote author="ian"] Let's get you SVN access so you can keep your progress in a separate folder or something. ... Here's the info...[/quote]
I'll read up on it and try to learn how to use SVN.
[quote author="ian"] I'm sure there's other people (me ) who would like to play along at home[/quote]
I'm not quite there yet. There are still some things I want to go through first because they will probably cause major structural changes. Like I just noticed that the SD code is setting up Time 1 but Timer 1 is then later configured by the FreeRTOS scheduler when it starts. So right now there are two different initializations for Timer 1 in two different sections of code. I need to go through this and figure out what FatFs needs. I will probably move them on to different timers. I have done this before with FreeRTOS, moved it to Timer 5 because Timer 1 is used by certain hardware peripherals like the ADC. Then the scheduler "tick" can be decoupled from the file system timing and appropriate intervals can be set for each independently.
So till I get through a few more things like this there is still likely to be large code changes.
I also need to figure out what to do about another licensing issue. My uart.c file has a lot of code in it from the Microchip peripheral libraries that come with the C30 compiler. Some time ago I took the relevant library functions, which were each in their own file, and consolidated them in to one file with some code I added. The library routines were written for the most general case (like the ability to deal with 9 bit data) that I knew I would never need so I cleaned them up a bit. Since everyone with the C30 compiler already has the peripheral libraries I can probably just pull the library functions back out of my file. The only downside is that you are back to the bulky generic versions -- that and I like to have ALL the relevant source in the project directory where I can see and touch it -- it also really bugs me that Microchip does not include the source for the C standard library they include. The other option is for me to rewrite the file from scratch. I have always just written code for myself before and so never worried about copyright or licensing issues. Any thoughts on this?
[quote author="mattcallow"] I'm still not sure about using the stack for LFN, since that would require a large stack for each task using FatFS. [/quote]
I'm not really sure yet either. We'll have to figure that out when we get to it. Since we have to write the ff_malloc() and free commands anyway we have a lot of flexibility.
[quote author="mattcallow"] Stack size was just a guess. [/quote]
Good guess then.
[quote author="mattcallow"] Maybe once you've posted your code, I'll try some of this to see if it makes the stack more stable.[/quote]
That would be great.
I also found there was a project setting somewhere with a flag something like _ICD2RAM = 1 or similar. Removing that also freed up some more memory. I don't know if you have an ICD2 or if that was left over from somewhere else. I think this flag should normally be enabled and disabled by MPLAB as you turn debugging on and off. I have an ICD3 and not an ICD2 so my MPLAB may not have known what that flag was for and assumed it was a user flag.
[quote author="mattcallow"] It seems a shame to create a fork of the existing code, but so be it. [/quote]
Agreed. I do not necessarily want to create a fork (I just threw it out as an option) which is why I am just going to call this a personal project and offer anything I write back to the forum. I guess all I am really saying is that I do not want to try to back-port (or whatever it's called) my changes in to the SVN version because my code is now so different. If you want to merge any of my stuff in great, if not don't worry about it. If you like what I end up with enough you could possibly even switch the code in SVN to it if you felt so inclined. I have never used SVN other than for just downloading code so I don't know how it would deal with the large structural changes I've made.
[quote author="mattcallow"] I think it may be possible to reduce the heap size. I don't know much is really needed. The reason for using heap_3.c rather than either of the other heap models is an effort to consolidate the dynamic memory allocation schemes in use, which (i think) will lead to better use of the available memory. Currently, the microchip libraries (e.g. printf) will use dynamic memory (i assume via malloc/free), uIP has it's own memory blocks (used in the telnet server), FreeRTOS needs to use one of heap_1/2/3.c and FatFS will need dynamic memory for LFN. Using the heap as the underlying dynamic memory pool for all these schemes seemed like the best way to go (in terms of memory usage).[/quote]
In the last couple hours I started bumping up against the the memory limit so I began investigating the issue. It turns out uIP does not use any dynamic memory (as you noted above). It statically declares all the buffers it needs at compile time. FatFs also does not use dynamic memory unless long file name (LFN) support is turned (as you also noted). Even with LFN support enabled it looks like it has an option to just use the stack. So FatFs my never need dynamic memory. If it does you need to write its ff_malloc() for it anyway. I switched FreeRTOS to heap_1.c (which is what I have always used) since FreeRTOS never frees memory unless you delete a task or a queue -- and I can't think of a case where you would ever want to do this. This let me set the compiler heap to zero. You can't disable it or the linker will complain if you use any stdio.h input or output functions. But if you never need a FILE * pointer (like if you only use printf() to a serial port) you can set the heap to zero.
I also noticed that the the configMINIMAL_STACK_SIZE was set to 256 bytes and the uIP task was using twice this value. This is probably twice what most tasks need but it turns out it was a good value for the uIP task. Was this a guess or did you measure it? I set the configMINIMAL_STACK_SIZE back down to 115, the default for the dsPIC/PIC24 port, but created a configUIP_STACK_SIZE of 512 for the uIP task. I measured the uIP stack grow to a max of 498 so 512 is probably a good value for now. We'll need to keep an eye on this and see if it gets any closer to the top of the stack. With the configTOTAL_HEAP_SIZE set at 2048 and the uIP task and two other tasks (the idle task and my console task) of default stack size there are still 310 bytes left in the FreeRTOS heap -- probably enough for two more normal sized tasks. Of course the configTOTAL_HEAP_SIZE could just be brought down to make this go to zero if desired since once all tasks and all queues are created no more memory is ever needed.
All this brought the total memory usage down to about 79% from what ever it was before -- ninety-something I think.
Now that I have some breathing room I'll get back to what I was working on.
[quote author="rhyde"] How is this effort proceeding? [/quote]
I guess I should start by saying I failed miserably...
...that is I failed miserably at not making major functional changes and wholesale format changes at the same time.
The more I went through the code the more I found things that weren't quite right or were just ugly. (edit: I should have added "or were things that I just didn't like.") Trying to hack fixes in would only have made the situation worse. So I decided to forget about trying not to change things too much. I also used a bunch of code I wrote for other projects that does initialization and what not.
So other than completely rearranging the source files and directory structure I have:
Fixed the oscillator PLL switching. The original code was not unlocking the OSCCON low and high registers before writing them so the PLL was never changed from the reset default. The defaults would have it running at over 92MHz (80MHz Max) so I don't know if it was actually running overclocked or if it would somehow end up running slower since parts of the oscillator circuit may not have been able to keep up. I added a couple files (osc.c/h) I use for configuring the oscillator to make it cleaner.
The configuration bits were being set in two different files and were not quite correct. For example, the OSC2/RA3 pin was not set as a digital pin as is needed by the new v1.1 hardware. There may have been something other issue too -- I don't remember now. Again, I added a file (configbits.h) I use to keep this clean.
I cleaned up the peripheral pin select setup and moved it to separate files (pps.c/h & ppsdef.h) that use standard macros to make the configuration a little easier and more obvious.
I implemented the check for whether a USB host is attached and the FTDI chip is powered before enabling the serial port transmitter. There was a note about this in the source but it was not being done. Right now it only checks one time on start up and turns it on or off permanently. I will try to make this a dynamic check later.
The value of BRG was one count low. It was 0.9% high before now it is 0.2% low. Though either of these are well within spec.
I wrote a temporary replacement for the standard library write() function because the Microchip included version would always enable the UART transmitter. If I have it off because there is no USB host attached I do not want a call to printf() to turn it on. Eventually this will interface with the system logger. It seemed cleaner and easier to make the change once in write() instead of changing each call to printf() for now. Once I get to work on the logger I may want to do this differently but lots of code seems to just use a printf() to spit out debugging info so this might end up being the best solution anyway.
I added interrupt driven serial code with circular transmit and receive buffers. This took a little longer than it should have because the routines I had been using in my projects were copied out of a book and so I thought I should rewrite them from scratch for this. Every time I try to code a circular buffer I end up confusing myself and so have to work through the code with paper and a pencil to make sure all the index pointer math is correct. I have versions of putc() and getc() implemented so far.
This is where I am at as of right this minute. I am just now starting to write the console/terminal code to create the user interface. Once I get this working with the serial port I will also connect it to the telnet server.
[quote author="rhyde"] Is more help required? [/quote]
Right now I don't know what is going to happen with the code I am working on. Since Matt started this project I guess it is kind of his call what he does with it. I am working on this because I know I will use it myself. I will post it here once I get to a good stopping point and then people can look at it. It will not be able to be merged back into SVN because it is too different. Whether Matt wants to port any of my work to the SVN version is up to him. If anyone else actually wants to use it I guess we could call it a fork and see if Ian wants to add it to SVN. If Matt really wanted to I suppose he could use my version. Otherwise it will just be a personal project.
[quote author="rhyde"] I may have some time in a few weeks to help if it makes sense. [/quote]
By that point I should have a version or two of mine posted here. I don't know how much time Matt has to devote to this. I know he wants to get the stability of the web server fixed. That might be a good area to look at. Any work done on the uIP code would be useful to everyone. I don't expect to be mucking around in the uIP code if I can avoid it. With the possible exception of trying to figure out how much dynamic memory it really needs and reducing the heap.
[quote author="mattcallow"] ...but memory is quite tight at the moment...[/quote]
This was one of the things that struck me as strange the first time I compiled the project but I have not looked in to it too far yet. The uIP code seems to have a lot of (and many big) variables but I imagine this has already been scrutinized quite a bit by others already. The other big target is the heap. Do we need 3kB of heap space? I have not looked yet but what requires this much dynamic memory allocation?
Okay, I just took a quick look and you are using heap_3.c in FreeRTOS. I have always used heap_1.c before in my projects. What drove the need for true dynamic memory? Was it the uIP stack or the FatFs file system?
Okay, another quick look says they are both using it. I do not think this is something I will get too soon but but my guess is that they do not need that much. The amount needed by the file system should be very predictable based on how many files you need to have open at once (1 or 2?) and whether or not it is doing any buffering. The uIP code I have no idea about what it should actually need.
It appears that I2S is a relatively fast protocol with clock rates over two MHz so the Bus Pirate is not likely to be able to support this. If all you need to do is "sniff" the data then the OLS is more what you are looking for anyway. The Bus Pirate's strength is its ability to speak a protocol and interact with the bus not act as a logic analyzer.
[quote author="mattcallow"] You do know that there is a telnet server in the uIP code base, dont you? It's not brilliant but it does work.[/quote]
Yes, I read through the code to try to understand what it was doing but have not attempted to compile it or run it yet. It seems to be so basic that it is more of a capability demonstration than a functional telnet server but I could be wrong. I have to study the uIP data structures and API a bit more to fully grasp what all it is doing or not doing.
[quote author="mattcallow"] ...the existing telnet server ...the uIP implementation is not that good - it does not handle lots of output very well.[/quote]
That was my impression also which is what led me down the rabbit hole of trying to understand the telnet protocol -- but I have learned a lot.
[quote author="mattcallow"] Hardware CRC and 32bit math would be good.[/quote]
These should not be too hard but I would like to get the console working.
[quote author="mattcallow"] My comments on this: - Be careful when changing code which uses proto-threads/proto-sockets.[/quote]
I'll steer clear of the uIP code other than what I need to change in the telnet server. Overall it and the FatFs code looked pretty good. The FreeRTOS code is messy because they write it to support so many different processors. In my own projects I have done quite a bit of simplification of this code.
[quote author="mattcallow"] - Try to separate functional changes from refactory when commiting to svn.[/quote]
[quote author="mattcallow"] - Since this code is built up from lots of different components... I suggest that we try to keep the format the same as the existing files.[/quote]
Also sounds good.
[quote author="mattcallow"] This logger is my design/fault. The reason for using printf style function is to reduce memory. The reasoning is as follows: The logger runs in it's own thread. Therefore, it must copy have a local copy of all variables passed to it that it will use. Since they may go out of scope in the original thread. For this reason, the logger stores log messages on queue. If we store the whole message as a string, then we use up lots of RAM. My idea is to only store copies of the variables. The format string is a constant, so doesn't need to be copied. For example, consider logging the following:
"Sent %x bytes on port %dn", 40, 80
If we use char* for the logger, we would need to copy 25 bytes for the message. Using const char* and vargs, we only need to copy a pointer to the string (2 bytes) plus 4 bytes for the arguments (assuming they are shorts)
So we save 19 bytes of RAM per message.[/quote]
This method is only beneficial if the const format string is a significant portion of the output. The uIP logging function, uip_log, for example does not take advantage of this so, as you have had to do in uip_task.c, it will always be "%s", m. My guess is that most any other code that we use or convert will be expecting to spit its messages directly out to some serial port and so will not be able to easily take advantage of this without us having to rewrite each call instead of just one function but I could be wrong. I just don't think the added complexity will be worth the little memory savings we will see in actual use. Also, I have not studied what you have so far but each slot in the queue would have to be big enough to hold the longest string. This is a fixed length so there is no memory savings once the queue is created. I guess since the logger will be writing to the console asynchronously anyways both methods could easily coexist.
[quote author="mattcallow"] I don't really understand why you would want to log to both the serial port and telnet?[/quote]
Because if the telnet connection is the interface I am using to try to troubleshoot some higher level function or application I would like to be able to see the log/error messages that I would see if I was connected via serial console. And since the log messages will not be written to a file -- that is probably overkill -- I think the only easy way to get them is to (possibly optionaly) have them print to the telnet connection as well. The only limitation of the telnet connection is that it would not be available early in the "boot" process. That and if you crashed the network you might still have the serial console.
Speaking of crashing the network, I have found that many/most times if I request a nonexistent page the PIC will reset. Sometimes I get my 404 page but most times it hangs.
Web Platform Last reset was 0080 external SD Power on CardType=4 0 disk_initialize(0) returns 0 f_mount(0) returns 0 Reading SD Card... SD Power on CardType=4 fs_type=2 csize=64 NIC DMA Init nic init uip init arp init dhcp configured. My IP=10.0.0.134 opening /INDEX.HTM Calling generator fn Copied 229 bytes. Tried to copy 229 bytes Waiting for data to be sent. sendlen=229 Data sent. sendlen=0 Calling generator fn Copied 81 bytes. Tried to copy 81 bytes Waiting for data to be sent. sendlen=81 Data sent. sendlen=0 opening /a opening /404.HTM Calling generator fn Copied 211 bytes. Tried to copy 211 bytes Waiting for data to be sent. sendlen=211 Web Platform Last reset was 0000 SD Power on
In the case above I rest the PIC, requested 10.0.0.134/ and got the index.htm file as expected. I then requested 10.0.0.134/a and it tried to send the 404.htm file but it crashes and resets. The browser gets part of the 404 page because it changes the browser title but never displays the page -- the wheel just keeps spinning. I have not tried to look in to the cause of this.
[quote author="rsdio"] There are many different levels, and so there is not one perfect answer to your question.[/quote]
That's kind of what I figured.
[quote author="rsdio"] ...this assumes that the existing files are consistently formatted...[/quote]
...which they aren't.
[quote author="rsdio"] Theoretically, if you have a suggestion for improving the formatting style, and you can get agreement from the active participants, then you could solicit permission from the group to make your changes.[/quote]
I think Matt and I are the only ones working on this so I guess I just need to find out if he has any strong opinions.
I don't really have any strong preference. I have seen many different coding styles and they all have benefits and drawbacks. I can work with any standard as long as somebody picks one.
[quote author="rsdio"] However, I would suggest that this be done in one single massive submit, not slowly over several feature changes.[/quote]
I can easily see doing one whole file at a time as I work on them but I don't know about trying to do the whole project all at once.
[quote author="rsdio"] I think I know how you feel, though, because I often cannot resist global search and replace for annoying things like trailing white space on lines, inconsistent indentation, random white space before and after parenthetical operators, et cetera.[/quote]
[quote author="rsdio"] If you plan on submitting any of your changes, I suggest that you first write the group and list the sorts of things you plan on changing before making massive edits to the source.[/quote]
Well, after trying to layout how I was going to implement the serial console and telnet server I started doing some reading. Like a fractal, this seemed simple from a distance but the more I look into the details the more complicated it gets -- the devil is in the details as they say. My assumption that there was one mode of character echoing that a telnet connection operates in was wrong. There are at least five permutations of local and remote echoing and either one or both end may or may not support or request echoing from the other end. I'm reading through the relevant RFCs right now because I could not find any other concise description of how the protocol is supposed to work. While several of these go back to 1971 it turns out telnet is even older but wasn't really documented till then. The sample telnet daemon in uIP does not support switching modes so I have to work out how to handle it. At this point I'm not even sure what mode it is using or what it is expecting the other end to use.
Since I am only planning to have one shell instance there will be a lot of coupling between the serial console and telnet server code so I want to make sure I have it worked out in my head how it is all going to work before I start writing any code. Maybe by tomorrow I can actually start coding something.
I'm glad you had better results. I tried using it by converting a local copy of the Bus Pirate project that I have been working on from an MPLAB 8 to an MPLAB X project. Everything compiled just fine after a few minor adjustments like you noted. Unfortunately the IDE would not load the ELF file. There was a comment on the Microchip forum about someone else having a similar problem. It sounds like the developers found the problem and fixed it for the next release but gave no work around, patch or even a time frame for the next beta. I tried this on both a Windows and a Linux machine and got the same results.
I am glad Microchip is moving in the right direction but, for building on an open source IDE platform they still seem to be working with the mindset of a completely closed source project. There's no source code available, binary only installation even under Linux, no real information flow from what I have seen between the programmers and the users, etc. Hopefully they will get better at this. NetBeans seems to have a dual license that would permit them to keep their stuff closed source. Hopefully they will eventually release it.
That sounds promising but I haven't looked at the MIT class. My into to VHDL was many years ago in a rapid system prototyping class but I have not touched it since. If it was something I could experiment with using the OLS I already have as the development board I would be more likely to give it a try.
My suggestion then is to design the examples to be able to work using the Papilio One or the OLS as the target. I understand someone would not be able to use the OLS as a logic analyzer and a development platform at the same time but I think there are probably quite a few people for who the OLS is not their only LA so this would not be a problem.
I have now started working my way through the code.
I was originally thinking I would start with the telnet server but I think there are a few other things I want to do first. The first thing is to get the logger and console working a little better. I see you started writing a logging task that will accept log messages via a queue; I will try to get that going. I want to integrate the hardware CRC peripheral instead of the software routine that uIP uses. I also want to look at the 32 bit math code of uIP and see if there is a better way to implement it as well.
In the only software collaboration I have ever done I was writing some functions that someone else would be using. There was very little interaction other than to agree on the function arguments and return values. I just wanted to say this so you know I don't really know how software collaboration is supposed to work.
The first thing I usually do when I start learning code someone else wrote and adapting it for my own use is to go through it and clean up all the formatting, clean out all the dead and left over code, add comments, remove unneeded variables, eliminate unneeded function nesting, fix all the compiler warnings, etc. Basically just simplify and clean up the code. I think this is now called refactoring but I could be wrong. I don't know if this would be appreciated or considered rude on collaborative project like this one. I don't mind cleaning things up because it helps me learn the code as I go through it but if it is considered bad form I will leave things alone.
My first questions are about the logger. Why did you design the log_message function to take a variable printf style argument list instead of just a char pointer? This seems a little overkill for this application but you may have had a good reason. I would have the log_message function take a char pointer and then copy the string to a queue. I think the application writing the log message should take care of any string formatting it wants and the logger should just handle printable strings. Did you start writing this logger from scratch or were you adapting something else? Let me know if you have any strong opinions on how this is done.
I am also starting to think about how to incorporate the logger with both the serial console and a telnet session. I would like to have both running the same shell code. In something this small I do not think there is a need to handle multiple shell sessions so I am thinking that the serial console and a telnet session will interact with one shell thread. They will both see the same output and one command interpreter will handle input from both at the same time. The character echoing and command buffers would have to be separate though since this will be local for the serial console and handled by the user's terminal for telnet. Let me know if you have any thoughts on this.