It is common for applications to use CSV files for recording data such as performance values or execution logs. And there could be more than one CSV files which can give you a meaningful outcome only after their consolidation. In this example, we are demonstrating how to merge multiple CSV files using Python without losing any data.
A CSV file, as the name suggests combines multiple fields separated by commas. The first row contains the name or title of each column, and remaining rows contain the actual data values.
Merge Multiple CSV Files in Python
Below is the complete code to perform the merging of CSV files. We used a CSV format which records the performance data for web services. While using this example, you may need to modify the HEADER variable (with the name of actual fields) as per the format of your CSV.
It is essential to understand the header fields we’ve used in this example. It includes the following:
- Timestamp – The time of a web service request
- Client IP – The IP address of the client system making the calls
- Web Service – The name of the web service used
- Status – The return code reflecting success or failure
- GOOD – The total no. of successful requests
- BAD – The total no. of failure calls
import os csv_header = 'Timestamp,Client IP,Web Service,Status,Good,Bad' csv_out = 'consolidated.csv' csv_dir = os.getcwd() dir_tree = os.walk(csv_dir) for dirpath, dirnames, filenames in dir_tree: pass csv_list =  for file in filenames: if file.endswith('.csv'): csv_list.append(file) csv_merge = open(csv_out, 'w') csv_merge.write(csv_header) csv_merge.write('\n') for file in csv_list: csv_in = open(file) for line in csv_in: if line.startswith(csv_header): continue csv_merge.write(line) csv_in.close() csv_merge.close() print('Verify consolidated CSV file : ' + csv_out)
A quick wrap up – Merge Multiple CSV Files
In this example, we covered “How to Merge Multiple CSV Files in Python.” It doesn’t use any special Python package to combine the CSV files and can save you a lot of time from going through multiple CSV individually.