MicroPython Skill Builders - #10 Files

Welcome to the tenth instalment of our MicroPython Skill Builders series by Tony Goodhew, aiming to improve your coding skills with MicroPython whilst introducing new components and coding techniques - using a Raspberry Pi Pico!

Today Tony will demonstrate how to use files with a MicroPython using a Pico, including creating them and then retrieving and processing the data later on. It builds on the previous tutorial about strings.

What you will need

We assume that you have installed Thonny on your computer and set up your Raspberry Pi Pico with the most recent MicroPython firmware (UF2). If not, check out our Raspberry Pi Pico Getting Started Guide where this is covered in detail.

You will need:

• Raspberry Pi Pico with header pins
• Micro-USB cable (for power and programming the Pico)

Optional (for the 'things to try' section at the end):

• 10K Ohm potentiometer
• DHT11 or DHT12 sensor
  
• Jumper wires (male/male and male/female, mixed colours ideally)

What is a file?

A file is collection of information primarily identified by its name, such as: data.csv or myfile.txt.

Why are files useful?

If you, for example, save a series of sensor readings in a list in your program, it is lost when the power to the Pico is removed. This is because Random Access Memory (RAM) is used to store variables as the program runs but, although it is very fast, it is also volatile and ‘forgets’ when the power goes off.

If we save the readings in a file in the Flash memory, we can still read it when we next power up the Pico.

Example 1 - Creating a File

In the example below we are going to create a simple Comma Separated Values (.csv) file. It is a plain text file with six records.

Each record is made up of two fields: an integer number (with no decimal point), and a floating-point number (with a decimal point). Each field is made up from a series of text characters.

Here is an example of such a text file as it appears when we print it in Thonny.

736,2247.38
547,6768.42
837,8634.73
415,4498.85
165,6724.25
872,566.26

There are 6 lines (records). The comma on each line splits the record into two fields.

Each record ends with \n which is a newline character. This is invisible when we print, but it ensures the next record is on a new line.

To bring that to life, this is how the output would display if we swapped that /n newline character for just a regular character N:

736,2247.38N547,6768.42N837,8634.73N415,4498.85N165,6724.25N872,566.26N

Such a file can be very long. The Flash memory of a Raspberry Pi Pico is 2MB – 2 MegaBytes – 2 million bytes! A byte is 8 bits - binary digits, a 1 or a 0 (did you know that half a byte is a nybble!)

We are going to write a program to create a similar file and then extract and process the data from it.

Create the file

Run the code below in Thonny:

# Data File Example
# Tony Goodhew for thepihut.com 25 July 2023
import random # Random Number Library - Built-In

print("\n#### Basic FILE Test ####\n")

# Part #1 - Create a .csv file
print("Values Recorded in file \n")   
with open("/test.csv", "w") as f: # w = overwrite if already there / a = append to end of file
    for count in range(6):
        irand = random.randint(0, 999)       # Generate a random integer
        rand = random.randint(0, 999999)     
        fp = rand/100.0                      # Generate a random FP number
        record = str(irand) + "," + str(fp)  # Build record string
        f.write(record + "\n")               # Write record to file with RTN/newline
        print(record)
    f.close()      

It produces output similar to below and a new file appears in the Raspberry Pico window at the bottom left of the screen in the Files section.

• No files section? Select View > Files to show that pane in Thonny.
• No file showing? You may need to refresh the view to see the newly created file (click the little menu 'burger' in the Files section)

#### Basic FILE Test ####

Values Recorded in file 

949,5474.87
202,5445.35
416,2676.86
564,7689.08
59,4670.19
558,3469.9

How does the file creation code work?

We create the new file with the instruction:

with open("/test.csv", "w")

The "w" specifies write mode and will overwrite an existing file with the same name, if it already exists. Using "a" instead would append any new material to the end of the existing file or create a new one if it was not found.

We set up a counted loop to loop 6 times. In the loop we use the random.randint(0, n) function to generate a couple of random integers of different sizes. We divide the second one by 100.0 to turn it into a floating-point value, fp.

We then build the file record converting the numbers to strings, and concatenate them with a separating comma character to make the record:

record = str(irand) + "," + str(fp)  # Build record string

We then write the record to the file adding the essential new line character to the end:

f.write(record + "\n") # Write record to file with RTN/newline

Finally, we print the contents of the record to the Shell window so that we can see what is going on.

When the loop finishes, we close the file with the instruction:

f.close()

Example 2 - Processing File Data

We're now going to process the data in the file we created above.

Add the lines below to the end of the file above then run it in Thonny:

# Part #2 - Process the file by first copying it into memory
print("\nProcessing lines from file: Method #1 - input whole file first\n")
with open("/test.csv", "r") as f: # Open the file for reading - "r"
    lines = f.readlines()         # Read whole file into memory
    print(len(lines),"\n")        # Print number of lines stored
    for line in lines:            # Get the next line from RAM 
        length = len(line)        # Find length of the current line
        line = line[:length-1]    # Remove last character - New Line
        p = line.find(",")        # Find position of the comma
        first = line[:p]          # Characters up to comma
        i2 = int(first)           # Convert to integer
        last = line[p + 1:]       # Characters after the comma
        fp2 =float(last)          # Convert to floating-point number
        result = i2 * fp2         # Process the values = Calculate a dummy result
        print("Int =",i2,"\tFP =",fp2, "\tAnswer =",result) # Tabbed printing

    f.close()  # Close the file when finished

How does the processing code work?

We print the title line and open the file in READING mode, "r"

with open("/test.csv", "r") as f: # Open the file for reading - "r"

The next instruction reads the whole file into RAM, splitting it into a series of individual lines at each new line character:

lines = f.readlines()         # Read whole file into memory
print(len(lines),"\n")        # Print number of lines stored

The print proves that there are 6 lines stored.

We then loop through the lines one at a time carrying out the following operations:

• Find the length of the current line: length = len(line)
• Remove the new line character from the end: line = line[:length-1]
• Find the position of the comma in the line: p = line.find(",")
• Use the position to extract the string in front of the comma: first = line[:p]
• Convert the string to an integer: i2 = int(first)
• Use the position to extract the string after the comma: last = line[p + 1:]
• Convert the string to a floating-point number: fp2 =float(last)
• Process the numbers by multiplying them together: result = i2 * fp2
• Print out the numbers and the product using: "\t" to tabulate with a Tab character

We finally close the file when we have processed all the lines using f.close()

After running the program, check that the numbers we saved in the first part are accurately recovered and processed in the second part.

Running out of RAM!

Change the loop counter from 6 to 100 and run it again. It should work fine and is pretty quick even with all that printing.

Now try changing the looping to 10000. Yes, ten thousand! The Pico has plenty of Flash memory to handle this. Now run the code again - this takes a good deal longer but it should successfully store all of the records, however the processing may fail due to a lack of RAM.

Unfortunately, there is not enough RAM to hold all the lines at once as it tried to read and store the separate lines.

RAM limitation processing workaround

We need a different method to process this large file and work around our RAM limitation. It's very simple too - we just need to read the lines in one at a time and process as we go.

Swap out the processing code (Part #2) with the block below (Part #3) and try again. It takes time but it gets to the end without failure:

# Part #3 - Process the file by reading one record at a time from the file in ROM
with open("/test.csv", "r") as f: # Open the file for reading - "r"
    print("\nProcessing lines from file: Method #2 - Input single lines\n") # Read one line at a time
    line = f.readline()           # Read in characters to make a line which ends with New line character from ROM/Flash memory
    while line != '':             # Empty string - Nothing left to read - End of File found
        length = len(line)        # Find length of current line
        line = line[:length-1]    # Remove last character New Line
        p = line.find(",")        # Find position of the comma
        first = line[:p]          # Characters up to comma
        i2 = int(first)           # Convert to integer
        last = line[p + 1:]       # Characters after the comma
        fp2 =float(last)          # Convert to floating-point number
        result = i2 * fp2         # Process the values = Calculate a dummy result
        print("Int =",i2,"\tFP =",fp2, "\tAnswer =",result) # Tabbed printing
        line = f.readline()

    f.close()  # Close the file when finished

Things to try

• Put a counter into the loop in the last piece of code and check that it does read all 10000 records
• Connect up a potentiometer and record the values obtained as you twist it for 30 seconds
• Read the temperature and humidity from a sensor and store 20 records at half minute intervals
• Read them back and print them out neatly (only one decimal place)
• Read them back and calculate the mean, maximum and minimum values of temperature and humidity
• Store the current time at the start of each record

Here's a hint for that last one...

import time
for i in range(10):
    now = time.time() # Seconds since epoch
    print(now)
    time.sleep(1)  

About the Author

This article was written by Tony Goodhew. Tony is a retired teacher of computing who starting writing code back in 1968 when it was called programming - he started with FORTRAN IV on an IBM 1130! An active Raspberry Pi community member, his main interests now are coding in MicroPython, travelling and photography.