My house was built by someone who had access to some building materials. Every window is a different size because the owner was a do it yourself person and he used left over parts from different jobs. It is frustrating at times because he messed a lot of things up.
I think there are enough users in the forum that could help you find cheap parts.
You can get cheap LEDs off of ebay. If you buy parts by the hundreds or thousands, the cost comes down and you can resell them or trade.
A lot of the sites that carry Arduino are cheap because their audience is students and people without a lot of money. Seeed Studio comes to mind.
The MSP430 by Ti was $4.30 at launch. I bought two Stellaris boards from Ti on promotion when they were $5.00 each.
Pics can be bought for less than $2.00. The most expensive part will be the Pikit 2 or Pikit 3 programmer. I also know a site that has tutorials for Pic chips.
NXP has some ARM chips for .39 cents.
Protostack has some cheap AVR development boards.
You might want to check out:http://www.taydaelectronics.com/
Sparkfun has a resistor pack with 500 resistors of different values for around 7 dollars. I would have to check the exact price.
I have dealt with Futurlec but other people say to be wary of them. I did get some components cheap from them.
I bought an ARM board from Mouser for $16 but the Jtag connector was like $50 and I'm not looking forward to buying the ST Link. This is where electronics can get expensive because the development boards were just as expensive.
There are some electronic stores with junkbox electronics but I am not going to suggest going there.