How to Merge Multiple CSV Files in Python

It is common for applications to use CSV files for recording data such as performance values or execution logs. And there could be more than one CSV files which can give you a meaningful outcome only after their consolidation. In this example, we are demonstrating how to merge multiple CSV files using Python without losing any data.

A CSV file, as the name suggests combines multiple fields separated by commas. The first row contains the name or title of each column, and remaining rows contain the actual data values.

Merge Multiple CSV Files in Python

Merge Multiple CSV Files in Python

Merge Multiple CSV Files

Below is the complete code to perform the merging of CSV files. We used a CSV format which records the performance data for web services. While using this example, you may need to modify the HEADER variable (with the name of actual fields) as per the format of your CSV.

It is essential to understand the header fields we’ve used in this example. It includes the following:

  • Timestamp – The time of a web service request
  • Client IP – The IP address of the client system making the calls
  • Web Service – The name of the web service used
  • Status – The return code reflecting success or failure
  • GOOD – The total no. of successful requests
  • BAD – The total no. of failure calls
import os

csv_header = 'Timestamp,Client IP,Web Service,Status,Good,Bad'
csv_out = 'consolidated.csv'

csv_dir = os.getcwd()

dir_tree = os.walk(csv_dir)
for dirpath, dirnames, filenames in dir_tree:
   pass

csv_list = []
for file in filenames:
   if file.endswith('.csv'):
      csv_list.append(file)

csv_merge = open(csv_out, 'w')
csv_merge.write(csv_header)
csv_merge.write('\n')

for file in csv_list:
   csv_in = open(file)
   for line in csv_in:
      if line.startswith(csv_header):
         continue
      csv_merge.write(line)
   csv_in.close()
   csv_merge.close()
print('Verify consolidated CSV file : ' + csv_out)

A quick wrap up – Merge Multiple CSV Files

In this example, we covered “How to Merge Multiple CSV Files in Python.” It doesn’t use any special Python package to combine the CSV files and can save you a lot of time from going through multiple CSV individually.

If you’ve learned something from this example, then care to share it with your colleagues. Also, connect to our social media (Facebook/Twitter) accounts to receive timely updates.

Best,

TechBeamers