Dangerous Prototypes

An open source USB stack has been a bit of a while whale over the past few years, but it's finally about to happen!

The latest Honken-JTR stack is working great on the IR Toy, and has been integrated into Bus Pirate v4. You can find the IR Toy test version here:
http://code.google.com/p/dangerous-prot ... -freestack (http://code.google.com/p/dangerous-prototypes-open-hardware/source/browse/#svn%2Ftrunk%2FUSB_IR_Toy%2FFirmware-freestack)

In a few days there will be a big announcement on the blog, followed by a test release for the IR Toy. JTR also fixed a ton of IR Toy bugs, those changes will go in this release..

The plan:
1. Announcement, release candidate of IR Toy firmware and software
2. Test release of Bus Pirate v3 and v4 (new stack) firmware v6
3. Sjaak and I are going to work up a ds30-loader compatible bootloader for BPv4 using the CDC stack. It will be fat, quick, and dirty. After some testing, Bus Pirate v4 will probably get an initial production quantity.
4. Release stack development/demo package, documentation
5. Port the stack to the other USB projects (Logic Sniffer, Logic Shrimp, USB LCD backpacks, etc)

Yeah! I hope I still know my way around the bp sources :)

great news ... waiting for BP4 price to go to production, hope this will drop down the price ($50 are too much) so I can get hand on it and stard devs.

The price will certainly drop. This price was set high to discourage people to buy it unless they are real devvers.

I think Ian can say what the price will be really. I guess it will be almost the same as bpv3  We will still continue to support BPv3 BTW. There will be a couple of firmware you need to choose from instead of a all-in-one.

The price will certainly drop. This price was set high to discourage people to buy it unless they are real devvers.

Yup, we only had 20 made so this was to discourage people who just wanted to "be ready" for v4 (and would be disappointed by our glacial pace and total lack of support).  I refunded 50% of the cost (or more) to any developers who contribute code. The final price will probably be around $35-$40. The v3 will still be active and sold at the same price as it is now.

Thanks for these clarifications :)
Will order one these days from SeeedStudio. I have to order some PCBs and accessories from them in few days.

awesome news - so now it'll be much easier to do straight USB implementations on PIC's, without the intermediary FTDI converter - and not have to point to MC sources - i like it!

[quote author="ian"]
3. Sjaak and I are going to work up a ds30-loader compatible bootloader for BPv4 using the CDC stack. It will be fat, quick, and dirty. After some testing, Bus Pirate v4 will probably get an initial production quantity.
[/quote]

You #$@%*^*! That was my "surprise" Ian. (Except that my version was going to be lean, slow and clean...)

I have it done (well... except for a tiny, minor detail that it does not work, but hey it was for dangerous prototypes. :)

But otherwise, you have even got me excited.

Now *I'm* even more excited!

I "think* I have the bootloader "mostly" working. It talked to the GUI and managed to write over my bootloader code so that is something.

There is a problem in that the GUI expects the bootloader to be placed at the end of the code space. This is not so easy (for me) with the large PIC24FJ parts as the function pointers in C30 are only 16-bit and there are all sorts of hoops to jump through with "FAR" code, handles etc. Hoops I could not manage. So I put the booloader at the bottom of the code space. Unfortunately the ds30 GUI throws an index out of bounds of array exception as soon as I tell it the bootloader is 172 pages from the end.

At this point maybe I should hand what I have over to the experts and see if they can fine tune this. I just do not have the required knowledge of FAR mode code, C30 etc, etc to finish this up in a timely way.

What say ye experts?

I would say the best place for the bootloader is at the end of the chip. This way you don't really need a special linkerscript, rerouting interrupts, don't have small delays in isrs, etc..

I'm happy to take a peek at the code :)

[quote author="Sjaak"]I would say the best place for the bootloader is at the end of the chip. This way you don't really need a special linkerscript, rerouting interrupts, don't have small delays in isrs, etc..

I'm happy to take a peek at the code :)[/quote]

Hi Sjaak,

Thanks for your reply.  None of your above points seem to be an issue. I do not use any interrupts, IVTs or special linker scripts for the bootloader or the rewritten bootloader purposed USB stack.  The problem is that the ds30 Loader is bugged and therefore of no use unless it is fixed or the bootloader code is moved to the top of code space.

Now this is where I need help. The 100% open source ds30 loader in no longer 100% open source. The PC side C# code is no longer available!

If it were I could tinker with it until it worked. (JTR SOP).

That leaves option 2 and option 3.

My understanding is that because C30 uses 16-bit function pointers you have to have "handles" in lower code space thus defeating the whole point of moving it to upper memory. Perhaps I am wrong about this though. I could not make head nor tail of the C30 documentation in this regard. If you believe that this is possible then I am happy to send you the "alpha" code "as is" for you to see what you can do with it.

Failing this option 3 is for me to write my own PC side app that is not limited like the existing ds30 loader. However option 2 would be the easiest and quickest solution. If you can help let me know and that will free me up to work on other things we have got going.

Here is pirate loader, a simple console app that works with the same bootloader. I'll try to dig up my copy of the ds30 loader source, but I always used the one from the official site. The developer is active and he would probably be happy to get involved.

Thanks for your reply. None of your above points seem to be an issue. I do not use any interrupts, IVTs or special linker scripts for the bootloader or the rewritten bootloader purposed USB stack. The problem is that the ds30 Loader is bugged and therefore of no use unless it is fixed or the bootloader code is moved to the top of code space.

The bootloader doesn't use interrupts, but the program that gets loaded does. If the bootloader is located in the first flash block (together with all the reset vectors, they need all to be remapped. Thus the program (the user-program) needs a new linkerscript and remapped isrs.

Since we moved to the new bootloader placed at the end of the flash we had zero badflashed since. The old one did have a few quirks and some issues. (all bad flashes are related to uploading the wrong or wrongly bootloader installer)

My understanding is that because C30 uses 16-bit function pointers you have to have "handles" in lower code space thus defeating the whole point of moving it to upper memory. Perhaps I am wrong about this though. I could not make head nor tail of the C30 documentation in this regard. If you believe that this is possible then I am happy to send you the "alpha" code "as is" for you to see what you can do with it.

You are not right about jumping to the bootloader. the first two programword are used as resetvector:

0x000000 goto [0x00000001]
0x000001 jump adress (23-bit wide)
0x000002 isr1 vector
0x000003 isr2 vector
..

See also chapter 4.1 of the datasheet of the pic24fj256gb106. (this is a farjump). The compiler does use regular jumps (+32 or -32k) but there is an option for large data and large code that will cope with long jumps.

Hi Ian and Sjaak,

I have emailed the source code to Ian. I am just to spent to push on with this tonight. Time for some music and sleep.

Note that the code is for the 128GB106 as I am using an early prototype BPv4. Remember to change it and the linker script to 256GB106 if applicable.

Did the IR Toy v2 and Bus Pirate v4 (final) hardware not arrive? I can check any tracking I have for it if you like. It has been long enough they would replace it.

[quote author="Sjaak"]This price was set high to discourage people to buy it unless they are real devvers.[/quote]
I don't mind the v4 price too much, only $10 or $15 above final price.

But my question is this: Who gets the 'extra' profit? Does the money all go to Seeed, or does Dangerous Prototypes get some?

I still don't have the Bus Pirate, but some of the features seem like they might be useful to me to have on my bench. I don't mind making a +25% donation to Dangerous Prototypes : "for the cause" (spoken with Zorg's accent)

The new hardware is a bit more expensive to make. We'll also get a bit more profit, which makes it easier to deal with distributors. Some of the extra margin will need to cover the USB ID we'll have to eventually buy too.

I guess my goal wasn't clear: I want the extra money to go to Dangerous Prototypes.

So, should I wait to buy the new 'v4' because you'll have more margin? ... or is the current stock at Seeed exactly the same deal for you?

I'm fine buying now, even at $50, so you guys get a few extra ducats, but I can wait if it's better somehow for DP if I buy the 'production' version.

Since we only did 20 of the developer hardware, they are proportionally more expensive than the production hardware. DP will get about the same whenever you decide to buy.

Compiled bootloader is 0x1a70 bytes, needs 7 pages (1C00 bytes). PIC program space is 0x200 to 0x2abF8 (plus config bytes to 0x2abfe).

I think (think...I hate these) the bootloader can go at 0x29000 for 0x1BF8 bytes, then the config words take up the rest of the last page.

Memory map bootloader
  data  (a!xr)  : ORIGIN = 0x800,        LENGTH = 0x4000
  reset          : ORIGIN = 0x0,          LENGTH = 0x4
  ivt            : ORIGIN = 0x4,          LENGTH = 0xFC
  aivt          : ORIGIN = 0x104,        LENGTH = 0xFC
  program (xr)  : ORIGIN = 0x29000,        LENGTH = 0x1BF8
  config4        : ORIGIN = 0x2ABF8,      LENGTH = 0x2
  config3        : ORIGIN = 0x2ABFA,      LENGTH = 0x2
  config2        : ORIGIN = 0x2ABFC,      LENGTH = 0x2
  config1        : ORIGIN = 0x2ABFE,      LENGTH = 0x2


Except error:
Linker Script Error: section .handle must be allocated low in program memory.

So, with google, seems to be an issue:
http://www.microchip.com/forums/m179745-print.aspx (http://www.microchip.com/forums/m179745-print.aspx)
http://www.microchip.com/forums/m328414-print.aspx (http://www.microchip.com/forums/m328414-print.aspx)

Sjaak is way better versed in this, maybe he can look at it later.

I'm going to do the easy thing and go for results. I'll leave the bootlaoder at the beginning and try to get a PC app to work well with it. Then we'll add protections. If that flies, then maybe worry about relocating it.

As an aside - a reason we tried so hard not to waste that last page by protecting it was memory constraints. This time we have way more flash to waste (for now, lol)

[quote author="ian"]As an aside - a reason we tried so hard not to waste that last page by protecting it was memory constraints. This time we have way more flash to waste (for now, lol)[/quote]

640k is enough for everybody? :D

Working on getting the pirate-loader working with the USB bootloader now. It is identifying now :) Check the screenshot. Now for some data :)

---

[quote author="ian"]Did the IR Toy v2 and Bus Pirate v4 (final) hardware not arrive? I can check any tracking I have for it if you like. It has been long enough they would replace it.[/quote]

Sure they arrived and I did confirm this with you. I could not have done the IR TOY 2 stuff without the hardware. I'm not that good. The reason I'm using the old BPv4 is that it has a R/A programming header. I don't have any here for the "final" version but I am expecting some next week, along with a soldering/desoldering (hot air) station that was on special for today only. Less than half price. :)

Now if only the final BPv4 had a usb bootloader...

[quote author="ian"]Working on getting the pirate-loader working with the USB bootloader now. It is identifying now :) Check the screenshot. Now for some data :)[/quote]

Great, I got that far with the ds30 loader and even managed to bootload in some code. Right over the top of the bootloader itself.

Getting up to date on new code is always painful :)

In this section I noticed an erase that bypasses the erase request, so I commented it out:

    if (!errflag) {
#ifdef HAS_PAGES

        if (bootstruct.enableerase) {
            NVMCON = 0x4042; //Erase page on next WR

            __builtin_tblwtl((unsigned int) fulladdress, 0xFFFF);
            __builtin_write_NVM();
            while (NVMCONbits.WR == 1);
            bootstruct.enableerase = 0;
        }
#endif
/*
#ifdef HAS_PAGES

        NVMCON = 0x4042; //Erase page on next WR
        __builtin_tblwtl((unsigned int) fulladdress, 0xFFFF);
        __builtin_write_NVM();
        while (NVMCONbits.WR == 1);

#endif
*/

1. On my compiler, the address needed more () here or it always erases from 0x00:
        fulladdress = ( ((bootstruct.addrU) << 16) + ((bootstruct.addrH) << 8) + bootstruct.addrL);

2. Got that working and confirmed erase is working, yeah!

3. Now can't program. We send first block of data (after bootloader region) and it hangs forever. Verify shows no write or damage to the firmware. Debugging that now.

[quote author="ian"]Getting up to date on new code is always painful :)

In this section I noticed an erase that bypasses the erase request, so I commented it out:
[/quote]

The ds30 loader does the same thing. The code should be there.

1. On my compiler, the address needed more () here or it always erases from 0x00:
        fulladdress = ( ((bootstruct.addrU) << 16) + ((bootstruct.addrH) << 8) + bootstruct.addrL);

Have adopted the same code.

2. Got that working and confirmed erase is working, yeah!

3. Now can't program. We send first block of data (after bootloader region) and it hangs forever. Verify shows no write or damage to the firmware. Debugging that now.

Yeah, thinking about it, I did not write any code either. It was just an erase from 0x0000. I will look at the code myself. How about you send mw whatever PC app you are using? I could not recompile the pirate loader because I did not have "cl" in my path or on my PC. I don;t even know what "cl" is.

This is really frustrating. I have the correct data in the buffer on the PIC, and the correct CRC, but the PIC freezes on the for loop after        NVMCON = 0x4003; (is that right? it is 0x4001 in ds 30 loader).

----

Update: Turns out my compiler doesn't like the        for (i = 0; i < bootstruct.datasize;);, so I changed it to        for (i = 0; i < bootstruct.datasize;i=i+3); and it works ok (some other changes needed in the loop too...)

---

Ok, now we get through everything ok, but there is a verify error (56), trying now to sort that.

Thanks JTR,

I'll upload my current packages in a second. I'm using codeblocks pluss (GCC/windows IDE).

codeblock is free/GPL BTW. attached.

Single stepping I notice:
*there is a HUGE overhead on the for loops
*reason is training ; after        for (i = 0; i < bootstruct.datasize;i++); - it just sits forever, then verify starts with i-193 and fails

Fixed that in both places, now verify fails because all is 0xffffff. Will check to see if we are writing anything at all first, then followup with verify.

[quote author="ian"]This is really frustrating. I have the correct data in the buffer on the PIC, and the correct CRC, but the PIC freezes on the for loop after        NVMCON = 0x4003; (is that right? it is 0x4001 in ds 30 loader).

----

Update: Turns out my compiler doesn't like the        for (i = 0; i < bootstruct.datasize;);, so I changed it to        for (i = 0; i < bootstruct.datasize;i=i+3); and it works ok (some other changes needed in the loop too...)

---

Ok, now we get through everything ok, but there is a verify error (56), trying now to sort that.[/quote]


NVMCON = 0x4001;  is correct. I [s:]copied[/s:] researched the microchip hid bootloader but looking at it closer they use the word program mode. We want the row program mode so 0x4001 it is.

Problem is no actual write, checking it now.

I changed NVMCON, but it didn't help. Readback shows no new data in the programed range. Verify shows original firmware is still intact. I will single step some more. My latest project attached.

Me thinks that:
            __builtin_tblwth(offset + 1, dataword);

should be:            __builtin_tblwth(offset, dataword);

Same change for verify part too.

dataword = bootstruct.data;
            i++;
            dataword |= (bootstruct.data << 8);
            __[s:]builtin_tblwth(offset, dataword);[/s:]
            offset += 2;

dataword = bootstruct.data;
            i++;
            dataword |= (bootstruct.data << 8);
            __builtin_tblwtl(offset, dataword);
            offset += 2;

just noticed that too, will I still need the offset +1 in the tblwtH? (will test)

I guess not, makes sense, but still no writing

I beleive not. The same address is used for tblwtl and tblwth

What we appear to need is:

        TBLPAG = (BYTE) (fulladdress >> 16);
        NVMCON = 0x4042; //Erase page on next WR
        __builtin_tblwtl((unsigned int) fulladdress, 0xFFFF);
        __builtin_write_NVM();
        while (NVMCONbits.WR == 1);

got it. Still one word offset, tracking that down now. Offset needed to be +1 not +2.

        NVMCON = 0x4001;
        for (i = 0; i < bootstruct.datasize;i++)
        {
            dataword = bootstruct.data[i];
            __builtin_tblwth(offset, dataword);
            i++;
			dataword = bootstruct.data[i];
			i++;
			dataword |= (bootstruct.data[i] << 8);
			__builtin_tblwtl(offset, dataword);
			offset +=1;
        }

You mean that it is work now???

I apologize from stepping in the mid of dev cycle but what's the reason to move bootloader high? You still really want to use linker script and moving it high for a single mcu works but if later you want to use one with more space you need to go trough trouble of moving it again ... I don't see nothing wrong with bootloader sitting on beginning of prog space and a linker script following it up..

as for the first real usb stack for pic that is open source - way to go guys :)

It is the write that is offset, the first chunk is being lost. Here's the readout, it should start with 22 EF 43 (or something) and it is 3 bytes off of the data stream. The bytes in the buffer are ok, so I suspect a write problem.

---

It's overshooting in the last leg of the write loop. Trying different logic now.

[s:]usbbufgetbyte(&bootstruct.addrU);
            usbbufgetbyte(&bootstruct.addrH);
            usbbufgetbyte(&bootstruct.addrL);
            usbbufgetbyte(&bootstruct.cmd);[/s:]

            usbbufgetbyte(&bootstruct.addrU);
            usbbufgetbyte(&bootstruct.addrL);
            usbbufgetbyte(&bootstruct.addrH);
            usbbufgetbyte(&bootstruct.cmd);

*I think* Need to double check it but try it and see.

Fixed the loop overshoot, but still 3 bytes off (1 word...) Latest files, Sjaak is going to play now too.

Scrub my previous post. The original is correct.

The address is correct, it's soemthing abot writing. I'm stepping it now

[s:]for (i = 0; i < bootstruct.datasize;i++)[/s:]

        for (i = 0; i < bootstruct.datasize - 1;i++)

prog and verify.

Yes, that is what I did, stopped the overshoot of the array, but still getting everything offset by a word. First word occurs twice now it seems.

Offset should be +2 after all. Now for a full test:

        offset = (unsigned int) fulladdress;

        NVMCON = 0x4001;
        for (i = 0; i < (bootstruct.datasize-1);)
        {
            dataword = bootstruct.data[i];
            __builtin_tblwth(offset, dataword);
            i++;
			dataword = bootstruct.data[i];
			i++;
			dataword |= (bootstruct.data[i] << 8);
			i++;
			__builtin_tblwtl(offset, dataword);
			offset +=2;
        }
        __builtin_write_NVM();
        while (NVMCONbits.WR == 1);
        NVMCONbits.WREN = 0;

        offset = (unsigned int) fulladdress;
        bootstruct.blreturn = 'K';

        for (i = 0; i < (bootstruct.datasize-1);)
        {
            dataword = __builtin_tblrdh((unsigned int) (offset));
            dataword &= 0xFF;
            if (dataword != bootstruct.data[i])
                bootstruct.blreturn = 'V';
            i++;
            dataword = __builtin_tblrdl((unsigned int) offset);
            if ((BYTE) dataword != bootstruct.data[i])
                bootstruct.blreturn = 'V';
            i++;
            if ((BYTE) (dataword >> 8) != bootstruct.data[i])
                bootstruct.blreturn = 'V';
            i++;
            offset +=2;
        }

Ok it "works", no verify errors.

However:
1. erase seems to miss still, the first block, and random blocks of data were deleted
2. Data seems to save all over the place, big blank chunks, not sure if from erase or write errors. Probably not write because verify works fine for the whole chip.

We're in the IRC channel working on this:

ian____   I think it is the second erase at 145, testing
   ian____   that made a bunch of difference. Now I need to compare to actual dump

Congratulations everyone :) It works! Compared a 100% bootload to a clean dump, 100% match. Now for some protections and a little more testing. I'll post in a second.

ian____   The current bootloader protection is if (!((fulladdress > 0x2000))) {
   ian____   I guess it needs three things: 1)allow 0x00 to 0x200, but force re-write the jump vector
   _Sjaak   fulladres >= 0x2000
   ian____   2) protect bootloader region
   ian____   3) protect config words (last page)

Yeah, I goofed. I could NOT understand why there was a second erase now I do. I misread the compiler directive as #ifdef blah blah when it was in fact #ifndef blah blah. Glad I got the blah blah right at least...

That part of the code can be removed. As can all HAS_PAGES conditions. All the PIC24s that can be supported all have pages so that is redundant check.

_Sjaak   a row is 64 words
   ian____   but I don;t know words->bytes>rows>pages without writign it down
   _Sjaak   and a page is 512
   ian____   that sounds right, the buffer is 1 row/192 bytes
   _Sjaak   
   _Sjaak   relief
   ian____   8 rows per page are 512 bytes page
   ian____   but, we get to use the full address int he source, so we don;t really need to cound pages except 1 from the end
   _Sjaak   512 word!
   _Sjaak   
   _Sjaak   luckily after this you don't need to rememer it
   ian____   sorry, 8 rows/page * 64 words/row = 512 words/page
   _Sjaak   np
   ian____   flash is at 2abf6, so the last page is at -0x200 words
   ian____   2abff - 0x200 = 0x2a9ff. So the last page with the config words starts at 0xaa00?
   ian____   0x2aa00?
   _Sjaak   sounds ok


Here's a protection function I came up with for testing. I'll run it and then upload the code:

#ifdef PROT_BL
    if (fulladdress == 0x000) {//protect jump vector
        bootstruct.blreturn = 'P';
        errflag = 1;
    }else if ((fulladdress<0x2000) && (fulladdress>=0x200)) {//protect config words
        bootstruct.blreturn = 'P';
        errflag = 1;
    }else if (fulladdress >= 0x2AA00)) {//protect bootloader
        bootstruct.blreturn = 'P';
        errflag = 1;
    }

#endif  

Yes but this is hardcoding the protection without regard for the different parts and sizes that are available and can be used with this bootloader.

Also, the updated pirate loader is incorrect with the part identifier. So to is the bootloader as it is hard coded for the 128GB106 but you are calling it a 256GB106 in pirate loader.

You're totally right, and I'll define it out once it is working. I just want to see it go for now :)

This is the latest bootloader. The protection is NOT yet working.

The .hex included with teh pirate-loader bootloader app has nothing in the beginning (jump, bootloader area), but it does have config words that overwrite and the protection is not tripped.. I'll try to tune the protection now.

#ifdef PROT_BL
    if (fulladdress == 0x000) {//protect jump vector
        bootstruct.blreturn = 'P'; 

How can you protect the GOTO when you have to erase page 0 so you can update the IVT?  Instead the bootloader must REPLACE the data at 0x000000 with a GOTO bootloader. This code is in the ds30 loader but I have not ported it across as yet. After 0x000000 is replaced no error is generated.

        errflag = 1;
    }else if ((fulladdress<0x2000) && (fulladdress>=0x200)) {//protect bootloader
        bootstruct.blreturn = 'P';
        errflag = 1;
    }else if (fulladdress >= FLASHSIZE- 0x200) ) {//protect CONFIGS
        bootstruct.blreturn = 'P';
        errflag = 1;
    }

#endif

The information you seek is in bootloader.h

FLASHSIZE - 0x200

Also, pirate loader has a bug in identifying the correct PIC24. But that can be sorted out when it is updated to support all the PIC24 family.

} while (inbyte != 0xc1);

    //  if (WaitInReady()) { //it's always ready, but this could be done better
    WaitInReady();
    cdc_In_buffer[0] = DEVICEID; //answer with device id
    cdc_In_buffer[1] = 4;
    cdc_In_buffer[2] = 0;
    putUnsignedCharArrayUsbUsart(cdc_In_buffer, 3);

Just did that :P I actually added the jump vector replacement code to ds30 loader, that was our own doing.

Thanks for tip on flash define.

Ok, compare notes.

        if (fulladdress == 0) {
            bootstruct.data[0] = 0x4;
            bootstruct.data[1] = (BYTE) (STARTADDRESS);
            bootstruct.data[2] = (BYTE) ((STARTADDRESS >> 8) & 0xFF);
            bootstruct.data[3] = 0x00;
            bootstruct.data[4] = (BYTE) ((STARTADDRESS >> 16) && 0xFF);
            bootstruct.data[5] = 0x00;
   
        }

Updated pirate loader C file with all IDs included.

Oops, I commented out the existing code for the GA002. I forgot that this is for legacy, non usb bus pirates. Just uncomment

//        case 0xd4:
//            printf("PIC24FJ64GA002n");
//            break;

Yours, as always, is better and more complete :)

    if (fulladdress == PROT_JUMP) {//protect jump vector
		bootstruct.data[0]=0x04; //replace with bootloader start address
		bootstruct.data[2]=(unsigned char)(PROT_BL_START);
		bootstruct.data[1]=(unsigned char)(PROT_BL_START>>8);
		bootstruct.data[3]=0x00;
		bootstruct.data[4]=0x00;
		bootstruct.data[5]=0x00;

I am having a problem writing to page 0. It seems to erase page 1 and 2 all the way to 0x400 and erases part of the bootloader

[s:]I think #1 and #2 need to be swapped of rthe jump vector, at least that's how Sjaak fixed my code.[/s:]

scratch that, it is wrong.

[quote author="ian"]scratch that, it is wrong.[/quote]

Are you sure? Sjaak's correction looks right to me.

The order is Upper, Low, High so the correct code aught to be:

      bootstruct.data[0]=0x04; //replace with bootloader start address Upper
      bootstruct.data[1]=(unsigned char)(PROT_BL_START); Low
      bootstruct.data[2]=(unsigned char)(PROT_BL_START>>8); High
 
Word two:
    bootstruct.data[3]=0x00; Upper
      bootstruct.data[4]= (BYTE) ((STARTADDRESS >> 16) && 0xFF); Low
      bootstruct.data[5]=0x00; High

pirate loader may be doing it. Remember that it expects the users goto to be in the page just below the bootloader. This has got to change as it will be putting it in the AIVT space.

Mr. ds 30 loader joined us in IRC and pointed out that pages at 0x400 not 0x200, that is why the bootloader is being whiped out when I erase the first page :) New code after one more test, it looks to be pretty nice now.

I also removed the goto/jump fix from pirate-loader, will do the same with your code.

Don't know why but it seems I cannot talk in the IRC. Please don't think I'm being rude.

[quote author="JTR"]Don't know why but it seems I cannot talk in the IRC. Please don't think I'm being rude.[/quote]

i have never been able to join the irc channel

Here is my end of day work. I had to back-track a little when an unknown change caused everything to fail. Note that the bootloader is now located at 0x400 because I guess that is how big a page is...

*working bootloader, needs: attention to location and page lengths, does not warn P about bootloader region protect, more and better defines, open on PGC/PGD jumper or continue to main program, return defined DEVICEID

*pirate loader with JTR's mod and jump fix disabled: NOTE: returns wrong device type with the current bootloader but still works.

The BPv4.hex included in the archive is a BPv4 rlease 6 firmware to test with the bootloader. It has stuff in all the protection zones.

Sorry about the IRC, we should work that out some time. We didn't think you were rude :) We still had a fairly active conversation here :)

I feel really good about where I left off, and I'll spend a good chunk of tomorrow making it all pretty (if someone doesn't beat me to it).

Back to work :) I'm going to clean up the small things listed above and do some more extensive testing. I'll try to make some video for the blog.

I have done some of that. Writing the config protection now. Will email you what I have so far.

Here is my newest pirate -loader:
*bootloader protection is treated as a warning that a section is skipped, instead of an error

Here's my latest firmware with merged changes from JTR.

There is a problem with the addressing. Bootloader protection is triggered for pages 65 (10400) to 72, then 134-136, and the config bit protection is not working... Probably a variable type issue.

The problem was the way we calculated the falsh address (which the compiler warned us about). I'm sure this would be better by forcing all the vars to long, but it works:

			//calculate flash address
        	//fulladdress = ( ((bootstruct.addrU) << 16) + ((bootstruct.addrH) << 8) + bootstruct.addrL);
        	fulladdress = (bootstruct.addrU); 
			fulladdress=fulladdress<<8;
			fulladdress+=bootstruct.addrH;
			fulladdress=fulladdress<<8;
			fulladdress+=bootstruct.addrL;

#ifdef PROT_GOTO
    if (fulladdress == 0x0000) {//protect jump vector
        bootstruct.data[0] = 0x04; //replace with bootloader start address
        bootstruct.data[1] = (unsigned char) (BLSTARTADDR);
        bootstruct.data[2] = (unsigned char) (BLSTARTADDR >> 8);
        bootstruct.data[3] = 0x00;
        bootstruct.data[4] = (unsigned char) ((BLSTARTADDR >> 16) && 0xFF);
        bootstruct.data[5] = 0x00;
        } // EDIT to add this
#endif
        
#ifdef PROT_BL        
        if ((fulladdress <= BLENDADDR) && (fulladdress >= BLSTARTADDR)) {//protect config words
            //bootstruct.blreturn = 'P'; //fail silently for now....
            errflag = 1;
        }
#endif        

#ifdef PROT_CONFIG  // best guess not tested yet
        if (fulladdress >= (FLASHSIZE - (2 * (PAGESIZER * ROWSIZEW)))) {//protect config page

            bootstruct.blreturn = 'P';
            errflag = 1;
        }
#endif

Separated protections."Current latest."

Bootloader va1. Works great, verified by dumping after bootloading.

*All JTR and Ian fixes
*all pins hiz in bootloader mode
*jump to user space added
*go to bootloader on PGC/PGD jumper
*fully protected (config words, bootloader, goto address)

TODO:
*Add bootloader version at fixed location in memory
*Add command to exit bootloader (reset)
*see how small it can get by reducing the program memory area in linker script (I got it quite small)

Here it is :) alpha 1 of Bus Pirate v4! If you have Bus Pirate hardware please please help test the bootloader.

Package:
*USB bootloader hex
*v4 firmware
*pirate loader windows utility
*source

Procedure
1. program bootloader with pic programmer
2. connect PGC and PGD and power up (or press reset)
3. USB should connect
4. Modify test.bat with the correct serial port
5. run test.bat to upload the firmware to the bus pirate
6. Reset.
7. open serial terminal and connect. it should be a bus pirate


Firmware:
*Previous BPv4 v6 firmware with Microchip stack
*relocated to 0x2000 for bootloader

Bootloader:
*user jump to bootloader at 0x1FF8
*bootlaoder version number at 0x1FF6
*protected
*Use PGC/PGD jumper to enter
*new 0xff reset from bootloader command. Delays don't seem to work, needs to be much longer

Pirate-loader:
(needs to be merged with JTR's changes!!)
*P protect errors are now treated as 'skips' instead of errors. Remember how the v2 bootloader always gave errors in 0x400 to 0x800? Same thing, only now we are just ignoring it and saying these are skips initiated by the bootloader and it is OK. It is the firmware responsibility  to locate in the right place. These will happen in the bootloader range (0x400 - 0x2000) and in the configuration bits (last page).
*Added command to exit bootloader, but it needs some work ont he firmware side

[quote author="ian"]Here it is :) alpha 1 of Bus Pirate v4! If you have Bus Pirate hardware please please help test the bootloader.

Package:
*USB bootloader hex
*v4 firmware
*pirate loader windows utility
*source

Firmware:
*Previous BPv4 v6 firmware with Microchip stack
*relocated to 0x2000 for bootloader

Pirate-loader:
(needs to be merged with JTR's changes!!)

*Added command to exit bootloader, but it needs some work ont he firmware side[/quote]

There may be a few tweaks here and there but this is close to final. Additional changes to pirate loader are only to support more devices and there is no hurry on this. The posted pirate loader is good to go. I have a slight change in the bootloader to fix the exit delay and this probably will be addressed tonight. (Ian, I will send you my latest with this in it so you can test while we are "running hot."

All in all this have been a fun project that just happened along out of schedule. :) Thanks to all who helped.

Congrats!

Will do some testing this weekend.

I'll do some testing this weekend too.

Just one question: Any procedures to follow for a Linux test? I can try pirate-loader.exe with wine first, then try to compile the source in Linux? Does that sound good?

Here's a more complete package with linux build scripts.

I tried this on my linux PC, but it didn't enumerate. This is all the /var/messages ddid outout:

Aug 13 14:09:27 hobbypc user.info kernel: usb 3-1: new full speed USB device using uhci_hcd and address 34

A normal BPv3:

Aug 13 14:11:40 hobbypc user.info kernel: usb 3-1: new full speed USB device using uhci_hcd and address 35
Aug 13 14:11:40 hobbypc user.info kernel: ftdi_sio 3-1:1.0: FTDI USB Serial Device converter detected
Aug 13 14:11:40 hobbypc user.info kernel: usb 3-1: Detected FT232RL
Aug 13 14:11:40 hobbypc user.info kernel: usb 3-1: Number of endpoints 2
Aug 13 14:11:40 hobbypc user.info kernel: usb 3-1: Endpoint 1 MaxPacketSize 64
Aug 13 14:11:40 hobbypc user.info kernel: usb 3-1: Endpoint 2 MaxPacketSize 64
Aug 13 14:11:40 hobbypc user.info kernel: usb 3-1: Setting MaxPacketSize 64
Aug 13 14:11:40 hobbypc user.info kernel: usb 3-1: FTDI USB Serial Device converter now attached to ttyUSB0

kernel:

Aug 13 14:44:31 hobbypc user.notice kernel: Linux version 2.6.33.2 (root@puppypc) (gcc version 4.4.3 (Ubuntu 4.4.3-4ubuntu5) ) #1 SMP Thu May 27 10:56:32 EST 2010

The modules usbserial and cdc-acm is loaded:

Module                  Size  Used by
cdc_acm                11464  0 
ftdi_sio               26466  0 
usbserial              21522  1 ftdi_sio
..
usbcore                91279  8 cdc_acm,ftdi_sio,usbserial,usblp,usbhid,uhci_hcd,ehci_hcd

I guess it is somewhere in my linux setup as I can see the usb descriptor is read in correctly (the info in /sys/bus/usb seems correct). Also the vreg led goes one after 1 second or so (PWR, USB are also lit)

Any further tips?

(moving to windowsXP now)

o/ More succes on WinXP :)

The upgrade went flawless (except for finding a valid .inf) I attached a working one to this post. That one should work for the bootloader and BPv4.

HiZ>i
Bus Pirate v4
Firmware v6.0RC (r572)
DEVID:0x1019 REVID:0x0001 (24FJ256GB106 A3)
http://dangerousprototypes.com

Hi Sjaak

Is that the BPv4 working with the new stack? The code I gav3 Ian to look at I mean. The code that had not been tested and not expected to work?

Or is it something else again?

It is the old firmware with the microchip stack, recompiled to work with the bootloader.

Ok, thanks. The restores the status quo...

[quote author="Sjaak"]o/ More succes on WinXP :)

The upgrade went flawless (except for finding a valid .inf) I attached a working one to this post. That one should work for the bootloader and BPv4.
[/quote]

The VID/PID currently in the bootloader is the standard microchip issued VID/PID for their CDC stack. We can change that or leave it as is. Now that we are departing from using the microchip stack do we have an issue with using there CDC stack VID/PID pair?

The same goes for the ported BPv4 firmware.

One thing going for us is we can indeed use the same VID/PID pair for the bootloader and the BPv4 firmware and therefore the same .inf file for each.

I don't think we can use their cdc pid/vid for the bootloader. Fortunately ian already has acquired two new vid/pids for the bpv4. These are microchip licensed ones, but in their usage conditions there is no mention about using their library only using their hardware (pic24fj256GB106)

See http://dangerousprototypes.com/docs/USB ... n_projects (http://dangerousprototypes.com/docs/USB_VID/PIDs_used_in_projects)

For beta testing it is OK.

[quote author="Sjaak"]I don't think we can use their cdc pid/vid for the bootloader. Fortunately ian already has acquired two new vid/pids for the bpv4. These are microchip licensed ones, but in their usage conditions there is no mention about using their library only using their hardware (pic24fj256GB106)

See http://dangerousprototypes.com/docs/USB ... n_projects (http://dangerousprototypes.com/docs/USB_VID/PIDs_used_in_projects)

For beta testing it is OK.[/quote]

Yes, I agree and we can change over to the new VID/PID. In fact I think I already have it in the ported BPv4 firmware.

Ok, here is my status report for Windows Vista (Ubuntu 11.04 will be soon)

- Bootloader installed after a bit of trouble (forgot how to use my PicKit 2, it's been a long long time)
- Vista recognized the device as "CDC test" and connected it to COM25
- During firmware installations, I saw some "skipped" parts, others are OK
- Got a nice message: "Firmware updated successfully :)!"
- A message popped up, device name is "Bus Pirate v4.0", selected "Locate and install software"
- Waited for eternity (Searching for Windows Update, wtf?!)
- Unplugged the USB cable after 15 mins
- Selected "Don't f*ck up with it!" this time
- Opened up Device Manager, selected update driver, for driver folder, I chose "Bus_Pirateinf" folder (in the latest zip file Ian posted)
- A message popped up, "Do you really want to install this driver?", selected "Hell, yeah!"
- After a couple of errors, and a few insults to my laptop, finally managed to install (COM26)
- Open up PuTTY, select port COM26, baud rate 115200
- Success, here is the log file:

=~=~=~=~=~=~=~=~=~=~=~= PuTTY log 2011.08.14 00:24:20 =~=~=~=~=~=~=~=~=~=~=~=

HiZ>i
Bus Pirate v4
Firmware v6.0RC (r572) 
DEVID:0x1019 REVID:0x0003 (24FJ256GB106 A5)
http://dangerousprototypes.com
HiZ>?
General					Protocol interaction
---------------------------------------------------------------------------
?	This help			(0)	List current macros
=X/|X	Converts X/reverse X		(x)	Macro x
~	Selftest			[	Start
#	Reset				]	Stop
$	Jump to bootloader		{	Start with read
&/%	Delay 1 us/ms			}	Stop
a/A/@	AUXPIN (low/HI/READ)		"abc"	Send string
b	Set baudrate			123
c/C	AUX assignment (aux/CS)		0x123
d/D	Measure ADC (once/CONT.)	0b110	Send value
f	Measure frequency		r	Read
g/S	Generate PWM/Servo		/	CLK hi
h	Commandhistory				CLK lo
i	Versioninfo/statusinfo		^	CLK tick
l/L	Bitorder (msb/LSB)		-	DAT hi
m	Change mode			_	DAT lo
o	Set output type			.	DAT read
p/P	Pullup resistors (off/ON)	!	Bit read
s	Script engine			:	Repeat e.g. r:10
v	Show volts/states		.	Bits to read/write e.g. 0x55.2
w/W	PSU (off/ON)		<x>/<x= >/<0>	Usermacro x/assign x/list all
HiZ>

I'll mess up with Ubuntu in a few hrs. Any special requests before that?

curious if the bp does show up as a serial port with your setup.

Bootloader v2. Yikes :)

.

OK, it is time for Ubuntu test:

- Bootloader installed on a Vista PC (just in case)
- Put jumper, connect and run: ls -l /dev/tty*, with BP v4 unplugged and plugged, I can see that the device is attached to /dev/ttyACM0
- Run: ./pirate-loader_lnx --dev=/dev/ttyACM0 --hex=../BPv4.hex, last 8 entries say skipped by bootloader and OK at the same time
- Remove jumper, connect and run: ls -l /dev/tty*, with BP v4 unplugged and plugged, I can see that the device is attached to /dev/ttyACM0 again
- Run: screen /dev/ttyACM0 115200, success, the output is:

HiZ>i
Bus Pirate v4
Firmware v6.0RC (r572) 
DEVID:0x1019 REVID:0x0003 (24FJ256GB106 A5)
http://dangerousprototypes.com
HiZ>

Thanks for that. It is very pleasing to see that the bootloader is working on non-windows platforms.

Checked out with Vista PC again, I am able to connect from the same port (COM26) and communicate without a problem. Hope this was helpful.

Sjaak, shall we check out your setup?

[quote author="JTR"]Actually, bootloader V1 Yikes!

You will see what I mean when you get it for testing. :)[/quote]

is it dfu?

[s:]Nope. It would be ready now except I "appear" to have found a bug from V1 that slipped through testing.[/s:]

Scrub that. "appear" was the right word. There is an apparent bug to the very keen eye, but it is only an apparition and not real. I modified pirate-loader to not return a message inferring that it had erased something that I should not and did not.

Hi,
Nice to see it working.
I got good results from a Windows XP SP3 box inside parallels on a Mac OSX Snowleopard.

HiZ>~
Disconnect any devices
Connect (ADC to +5V)
Space to continue
Ctrl
AUX OK
MODE LED OK
PULLUP H OK
PULLUP L OK
VREG OK
EEPROM
SCL OK
SDA OK
WP OK
ACK OK
ADC and supply
Vusb(5.05) OK
5V(5.00) OK
5V0 VPU(4.74) OK
ADC(0.00) FAIL
3.3V(3.27) OK
3V3 VPU(3.04) OK
Bus high
MOSI OK
CLK OK
MISO OK
CS OK
Bus Hi-Z 0
MOSI OK
CLK OK
MISO OK
CS OK
Bus Hi-Z 1
MOSI OK
CLK OK
MISO OK
CS OK
MODE, VREG, and USB LEDs should be on!
Any key to exit
Found 1 errors.
HiZ>

OK, there's one error just because I didn't have any jumper wires at hand.

Worked fine on my macbook, running Snow Leopard. (That is, running natively in snow leopard) I only had to modify the "build-unix.sh" as it was looking for pirate_loader.c in a non-existent directory "source".

Here is the terminal output of the first small section, before all the programming information.

Greeeg$ ./pirate-loader_mac --dev=/dev/tty.usbmodem00000001 --hex=../BPv4.hex 
+++++++++++++++++++++++++++++++++++++++++++
  Pirate-Loader for BP with Bootloader v4+  
  Loader version: 1.0.2  OS: Darwin
+++++++++++++++++++++++++++++++++++++++++++

Parsing HEX file [../BPv4.hex]
Found 87552 words (262656 bytes)
Opening serial device /dev/tty.usbmodem00000001...OK
Configuring serial port settings...OK
Sending Hello to the Bootloader...OK

Bootloader version: 4,06
Device ID [f1]:PIC24FJ256GB106

It got to "Firmware updated successfully :)!" So I'm assuming all went well.
The firmware also passed a self test with 0 errors.

Thanks for the feedback everyone. It's great to know it's going so well.

Attached is an updated pirate-loader with JTR's changes:
*Support for many different PIC types
*Maybe also works with Bus Pirate v3.5 bootloader (untested)

Latest source and compile attached, and in SVN.

Updated firmware on BP v4 without a problem. Went really smooth, just one thing: the shell script is looking into source folder, just have to remove it before trying to compile.

Connected to my BP v2go, here is the output:

tayken@tayken-eee:~/dangerous_prototypes/Bus_Pirate_v4/pirate-loader-r1324$ ./pirate-loader_lnx --dev=/dev/buspirate --hello
+++++++++++++++++++++++++++++++++++++++++++
  Pirate-Loader for BP with Bootloader v4+  
  Loader version: 1.0.2  OS: Linux
+++++++++++++++++++++++++++++++++++++++++++

Opening serial device /dev/buspirate...OK
Configuring serial port settings...OK
Sending Hello to the Bootloader...OK

Bootloader version: 1,02
Device ID [d4]:PIC24FJ64GA002

Still having trouble getting the booltoader to recognized by my linux distro.

I have loaded the module cdc_acm which should provide access to the bootloader but it won't bind to the bootloader. It will however bind to the BPv4.

I noticed there are slight differences in the usb-descriptor:

Bootloader:

T:  Bus=03 Lev=03 Prnt=04 Port=03 Cnt=01 Dev#= 12 Spd=12  MxCh= 0
D:  Ver= 2.00 Cls=02(comm.) Sub=00 Prot=00 MxPS= 8 #Cfgs=  1
P:  Vendor=04d8 ProdID=000a Rev= 0.02
S:  Manufacturer=Dangerous Prototypes
S:  Product=CDC Test
S:  SerialNumber=00000001
C:* #Ifs= 2 Cfg#= 1 Atr=80 MxPwr=100mA
I:* If#= 0 Alt= 0 #EPs= 1 Cls=02(comm.) Sub=02 Prot=00 Driver=(none)
E:  Ad=81(I) Atr=03(Int.) MxPS=  10 Ivl=64ms
I:* If#= 1 Alt= 0 #EPs= 2 Cls=0a(data ) Sub=00 Prot=00 Driver=(none)
E:  Ad=02(O) Atr=02(Bulk) MxPS=  64 Ivl=0ms
E:  Ad=82(I) Atr=02(Bulk) MxPS=  64 Ivl=0ms

BPv4:

T:  Bus=03 Lev=03 Prnt=04 Port=03 Cnt=01 Dev#= 13 Spd=12  MxCh= 0
D:  Ver= 2.00 Cls=02(comm.) Sub=00 Prot=00 MxPS= 8 #Cfgs=  1
P:  Vendor=04d8 ProdID=fb00 Rev= 1.00
S:  Manufacturer=Microchip Technology Inc.
S:  Product=Bus Pirate v4.0      
C:* #Ifs= 2 Cfg#= 1 Atr=80 MxPwr=200mA
I:* If#= 0 Alt= 0 #EPs= 1 Cls=02(comm.) Sub=02 Prot=01 Driver=cdc_acm
E:  Ad=82(I) Atr=03(Int.) MxPS=   8 Ivl=2ms
I:* If#= 1 Alt= 0 #EPs= 2 Cls=0a(data ) Sub=00 Prot=00 Driver=cdc_acm
E:  Ad=03(O) Atr=02(Bulk) MxPS=  64 Ivl=0ms
E:  Ad=83(I) Atr=02(Bulk) MxPS=  64 Ivl=0ms

I also notice the Vreg led turn on under linux, under windows it doesn't.

I checked several thing with tayken, like adding drivers and unloading them, tried an onboard usb port, with usb hub..

I think it could be the minor differences in the descriptor, but dunno for sure.. will try this some other day.

Any other suggestions?

Here is a bootloader compile that has the came Prot=01 and power settings as the BPv4 firmware. Maybe this one will work?

MxPS=  10 Ivl=64ms

I'm not sure what to make of this one.

        LOWB(CDC_NOTICE_BUFFER_SIZE),                           // wMaxPacketSize (low byte)
        HIGHB(CDC_NOTICE_BUFFER_SIZE),                          // wMaxPacketSize (high byte)
        0x40,                                                           // bInterval

The 10 is maxpacketsize and it seems reasonable, but I'm not sure about interval.

[quote author="ian"]Here is a bootloader compile that has the came Prot=01 and power settings as the BPv4 firmware. Maybe this one will work?

MxPS=  10 Ivl=64ms

I'm not sure what to make of this one.

        LOWB(CDC_NOTICE_BUFFER_SIZE),                           // wMaxPacketSize (low byte)
        HIGHB(CDC_NOTICE_BUFFER_SIZE),                          // wMaxPacketSize (high byte)
        0x40,                                                           // bInterval

The 10 is maxpacketsize and it seems reasonable, but I'm not sure about interval.[/quote]

This endpoint is a dummy and not used. binterval has no meaning for bulk endpoints.

[quote author="ian"]The 10 is maxpacketsize and it seems reasonable.[/quote]

The reason for 10 byte packet size is so that seldom used getLineCoding can be reported in a single short usb packet avoiding the use of a empty packet to signal end of transfer.

No that is not correct. GET_LINE_CODING is a standard class request and is returned on the control endpoint and is 7-bytes long. The 10-byte packet is for the unused LINE_NOTIFICATION endpoint. The CDC specification requires that it be implemented  even though it is not used (or even armed in the bootloader stack.)  The reason why it is 10-bytes is because that is the size of a full "Serial State Notification" packet. As this is the exact size of the transfer requested (when used) no ZLP is required to be sent afterwards.

You are right. I have apperently forgot things during paternity leave.

[quote author="JTR"]The 10-byte packet is for the unused LINE_NOTIFICATION endpoint. The CDC specification requires that it be implemented  even though it is not used (or even armed in the bootloader stack.)  The reason why it is 10-bytes is because that is the size of a full "Serial State Notification" packet. As this is the exact size of the transfer requested (when used) no ZLP is required to be sent afterwards.[/quote]
But wait: the USB specification says that if the package size matches the max endpoint size, then there must be a ZLP to signal the end. The typical solution for this is to make the max endpoint size 1 (or 2) bytes larger than the largest actual payload. That's the only way to avoid the ZLP. Are you sure that the "Serial State Notification" isn't 9 bytes or 8 bytes?

Maybe I'm getting confused with Isochronous payloads, where I've been doing most of my work, lately, but I thought this was a universal rule for USB.