Increase file system size on V2

[kevinjwalters]

The flash storage increased from 256 on the V1 to 512KB on the V2. The file system still appears to be limited to "there is approximately 30k of storage available" according to https://microbit-micropython.readthedocs.io/en/latest/tutorials/storage.html.

I ran out of space for wav files for a previous project. I'm now running out of space for code with the double whammy of not being able to use mpy (bbcmicrobit/micropython#524).

[kevinjwalters]

For anyone else facing code size issues, I have found that pyminify from https://pypi.org/project/python-minifier/ helps.

[microbit-carlos]

Thanks for the request @kevinjwalters!
Yeah, we should consider adding a compile time flag to expand the fs to all the flash left available. That way advance users can easily compile MicroPython and take advantage of the extra space.

That being said, in V2 we also include the Nordic SoftDevice and bootloader, so that adds an extra 150-ish KBs. Between that and the additional features the last time I checked the max fs size possible was around 60 Kbs, but I might be wrong about that figure.

[kevinjwalters]

A compile option is useful for a minority but won't be of much use to a typical MicroPython programmer. I'd still like this to be increased in size on the V2. Perhaps a compromiste value that's viable in the long-term can be chosen based on the roadmap and guestimate on bloat and additions over time of libraries, SoftDevice and bootloader? As part of the increase in interpreter code size, mpy support would be great, at least for code in the file system.

A documentation update with accurate values for V1 and V2 would be useful too.

[kevinjwalters]

I just noticed this was 32k briefly then got reduced to 24k in d261e87 with the comment

Because the firmware is overlapping with the start of the filesystem, and
there needs to be room to grow the firmware further.

[kevinjwalters]

@dpgeorge I noticed you reduced the file system space for the V2 MicroPython port from 32k to 24k in d261e87 . This leaves it as the same or very similar size to the micro:bit V1. I was surprised by the sized as the nRF52833 (512K) on V2 has twice as much flash as the nRF51822 (256K) on V1.

The performance (4x clock, FPU, etc) and extra memory (4x) give the V2 far more scope for more advanced programs. I've run out of disk space despite minifying my code and library code for one of my projects. I was hoping to get the file system enlarged to squeeze some more functionality in. I can't use mpy due to bbcmicrobit/micropython#524

[kevinjwalters]

I'm hitting some corruption issues discussed in https://github.com/orgs/micropython/discussions/18081

This seems like it could be problematic?

micropython-microbit-v2/src/codal_port/microbitfs.c

Lines 89 to 90 in 7c304ec

	/** Maximum number of chunks allowed in filesystem. 252 chunks is 31.5kB */
	#define MAX_CHUNKS_IN_FILE_SYSTEM 252

[kevinjwalters]

The codal_port has a dependency on the nrf port for the hedaer file that specifies the chunk size which is expressed as a 2^n power in MBFS_LOG_CHUNK_SIZE. A comment discuss the size limitation.

#ifndef MBFS_LOG_CHUNK_SIZE
// This property can be tuned to make the filesystem bigger (while keeping
// the max number of blocks). Note that it cannot (currently) be increased
// beyond 8 or uint8_t integers will overflow.
// 2^7 == 128 bytes
// (128-2) bytes/block * 252 blocks = 31752 usable bytes
#define MBFS_LOG_CHUNK_SIZE 7
#endif

1. If that's correct then 8 is a viable number and would allow (256-2) * 252 = 64008 bytes presumably with some sort of penalty for small files.
2. Is enlarging the uint8_t variables used for chunks to uint16_t a better idea?
3. Tangential issue: should codal_port be independent of the nrf port?

(bbcmicrobit/micropython#246 has some background on file system design)

---

[Added later]: binary images of file systems are created for hex files by https://github.com/microbit-foundation/microbit-fs which makes changing chunk variable size more fun.

[kevinjwalters]

I tried MBFS_LOG_CHUNK_SIZE of 8 with 48k file system and it corrupted itself after I replaced one file in the file system, this strongly suggests 8 is buggy.

[Added later]: turns out conclusion here was wrong, scroll down.

[kevinjwalters]

I now think issues with file system size changes and chunk size changes are likely to be related to issues described in #231

[jaustin]

Thanks for the request @kevinjwalters! Yeah, we should consider adding a compile time flag to expand the fs to all the flash left available. That way advance users can easily compile MicroPython and take advantage of the extra space.

That being said, in V2 we also include the Nordic SoftDevice and bootloader, so that adds an extra 150-ish KBs. Between that and the additional features the last time I checked the max fs size possible was around 60 Kbs, but I might be wrong about that figure.

@microbit-carlos just talking to the makers of https://github.com/NovaSeq/RobotPu and they'd also find this really useful. (Not using python.microbit.org) so I think worth looking at - but probably not until we've fixed the issue @kevinjwalters is looking at.

CC @NovaSeq

[microbit-carlos]

For context, the main reason the filesystem size was left to 20ish KBs was to match the same space available in micro:bit V1.

This is because in mixed classrooms, where students might be using a micro:bit V1 or V2 we wanted to prevent situations where the code would fit only for students using a micro:bit V2.

That being said, ideally we should be able to provide a way for more advance users to expand this filesystem space. A runtime flag could be possible, but would require larger changes, which is why I was thinking a compile time flag could be an easier way to be able to deliver this feature.

@kevinjwalters one question, living MBFS_LOG_CHUNK_SIZE to 7 works when the filesystem is expanded the the maximum possible at that configuration (30ish KBs)? Or does that fail as well?

[kevinjwalters]

The "just recompile it" is a barrier to use. Even for experienced users there's normally at least one annoying minor hurdle to get over for this. I hit something on linux and switched over to a Raspberry Pi where I'd have easier control over the disposable environment.

You could offer two binaries, one which is intended to limit MicroPython to the V1 world and the other that's more useful for the capabilities of the V2. It would be interesting to get some real world feedback on V1 vs V2 issues if you don't already have that. I suspect the main issue with V1 vs V2 is being able to differentiate the two when faced with a pile of them and needing V2 features like the microphone for something. There was some good thought behind the visual cues to aid this. V1 users will also hit MemoryError exceptions before V2 users. A more faithful crippled MicroPython would run at the same clock speed with artifically constrained memory too.

An alternative very simple approach would be to have a large filesystem whcih comes with a big file in it to cap normal file system usage at V1 size. Anyone who wants to use it all could simply delete that file.

I think MBFS_LOG_CHUNK_SIZE of 7 or 8 both work well within the discussed limits. I just got confused by the issue described in #231 which happens if you move or resize a file system and use the hex files from the build system on a device which has previously has MicroPython installed on it.