File Operations in Go: Read, Write, Append, Delete, Binary & Large File Chunking

Go's standard library (os, io, bufio, io/ioutil in older versions) gives you everything you need to work with files — from simple one-line reads to streaming gigabyte-sized files in fixed-size chunks. This guide walks through all of it, from beginner to advanced.

Table of Contents

1. Setup
2. Reading Files
3. Writing Files
4. Appending to Files
5. Deleting Files
6. Binary File Operations
7. Large File Batch/Chunk Processing
8. Copying Files Efficiently
9. Best Practices & Common Pitfalls

---

Setup

No external packages are needed — everything shown here comes from Go's standard library:

import (
    "bufio"
    "fmt"
    "io"
    "os"
)

Create a working folder and initialize a module if you want to run the examples:

mkdir go-file-demo && cd go-file-demo
go mod init go-file-demo

---

Reading Files

1. Read entire file at once (simplest way)

Best for small-to-medium text or config files.

package main

import (
    "fmt"
    "os"
)

func main() {
    data, err := os.ReadFile("notes.txt")
    if err != nil {
        fmt.Println("Error reading file:", err)
        return
    }
    fmt.Println(string(data))
}

os.ReadFile loads the whole file into memory. Avoid it for very large files (see Large File Batch/Chunk Processing).

2. Read line-by-line with bufio.Scanner

Ideal for logs, CSVs, or any line-oriented text file.

package main

import (
    "bufio"
    "fmt"
    "os"
)

func main() {
    file, err := os.Open("notes.txt")
    if err != nil {
        fmt.Println("Error opening file:", err)
        return
    }
    defer file.Close()

    scanner := bufio.NewScanner(file)
    lineNum := 1
    for scanner.Scan() {
        fmt.Printf("%d: %s\n", lineNum, scanner.Text())
        lineNum++
    }

    if err := scanner.Err(); err != nil {
        fmt.Println("Scanner error:", err)
    }
}

By default bufio.Scanner has a max token size (~64KB per line). For very long lines, increase the buffer:

scanner.Buffer(make([]byte, 0, 1024*1024), 10*1024*1024) // up to 10MB per line

3. Read with bufio.Reader (fine-grained control)

Useful when you need to read by delimiter or custom chunk size instead of by line.

package main

import (
    "bufio"
    "fmt"
    "io"
    "os"
)

func main() {
    file, err := os.Open("notes.txt")
    if err != nil {
        fmt.Println(err)
        return
    }
    defer file.Close()

    reader := bufio.NewReader(file)
    for {
        line, err := reader.ReadString('\n')
        fmt.Print(line)
        if err == io.EOF {
            break
        }
        if err != nil {
            fmt.Println("Read error:", err)
            break
        }
    }
}

---

Writing Files

1. Write entire content at once

Creates the file if it doesn't exist, truncates it if it does.

package main

import (
    "fmt"
    "os"
)

func main() {
    content := []byte("Hello from Go!\nThis is line two.\n")
    err := os.WriteFile("output.txt", content, 0644)
    if err != nil {
        fmt.Println("Error writing file:", err)
        return
    }
    fmt.Println("File written successfully")
}

0644 is the Unix file permission (owner read/write, others read-only).

2. Write using os.Create + bufio.Writer (better for large/streamed writes)

package main

import (
    "bufio"
    "fmt"
    "os"
)

func main() {
    file, err := os.Create("output.txt")
    if err != nil {
        fmt.Println("Error creating file:", err)
        return
    }
    defer file.Close()

    writer := bufio.NewWriter(file)
    defer writer.Flush() // IMPORTANT: flush before the function returns

    lines := []string{"Line 1", "Line 2", "Line 3"}
    for _, line := range lines {
        _, err := writer.WriteString(line + "\n")
        if err != nil {
            fmt.Println("Write error:", err)
            return
        }
    }
}

bufio.Writer buffers data in memory and writes to disk in batches, which is much faster than calling file.Write repeatedly. Always call Flush(), or your final buffered bytes may never reach disk.

---

Appending to Files

Use os.OpenFile with the O_APPEND flag combined with O_CREATE (create if missing) and O_WRONLY.

package main

import (
    "fmt"
    "os"
)

func main() {
    file, err := os.OpenFile("log.txt", os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0644)
    if err != nil {
        fmt.Println("Error opening file:", err)
        return
    }
    defer file.Close()

    _, err = file.WriteString("New log entry\n")
    if err != nil {
        fmt.Println("Error appending to file:", err)
        return
    }
    fmt.Println("Appended successfully")
}

Common os.OpenFile flags

Flag	Meaning
os.O_RDONLY	Read-only
os.O_WRONLY	Write-only
os.O_RDWR	Read and write
os.O_APPEND	Append to file instead of overwriting
os.O_CREATE	Create file if it doesn't exist
os.O_TRUNC	Truncate file to zero length when opened
os.O_EXCL	Used with O_CREATE, fails if file exists (useful for atomic "create-only" semantics)

You can combine flags with | as shown above.

---

Deleting Files

1. Delete a single file

package main

import (
    "fmt"
    "os"
)

func main() {
    err := os.Remove("output.txt")
    if err != nil {
        fmt.Println("Error deleting file:", err)
        return
    }
    fmt.Println("File deleted successfully")
}

2. Delete a directory and everything inside it

err := os.RemoveAll("temp_folder")
if err != nil {
    fmt.Println("Error deleting folder:", err)
}

os.Remove fails on non-empty directories. Use os.RemoveAll when you need recursive deletion — but be careful, it does not ask for confirmation.

3. Safe delete: check existence first

package main

import (
    "errors"
    "fmt"
    "os"
)

func main() {
    path := "maybe.txt"
    if _, err := os.Stat(path); errors.Is(err, os.ErrNotExist) {
        fmt.Println("File does not exist, nothing to delete")
        return
    }
    if err := os.Remove(path); err != nil {
        fmt.Println("Error deleting file:", err)
        return
    }
    fmt.Println("Deleted:", path)
}

---

Binary File Operations

Go treats all files as byte streams — "binary" just means you're not treating the content as text. This is the same API, just applied to []byte without string conversions, and often combined with encoding/binary for structured binary data.

1. Read/write raw binary data

package main

import (
    "fmt"
    "os"
)

func main() {
    // Write raw bytes (e.g., an image, a serialized struct, etc.)
    data := []byte{0x89, 0x50, 0x4E, 0x47, 0x0D, 0x0A, 0x1A, 0x0A} // PNG header bytes
    err := os.WriteFile("sample.bin", data, 0644)
    if err != nil {
        fmt.Println("Write error:", err)
        return
    }

    // Read it back
    readBack, err := os.ReadFile("sample.bin")
    if err != nil {
        fmt.Println("Read error:", err)
        return
    }
    fmt.Printf("Bytes: % X\n", readBack)
}

2. Copy a binary file (e.g., image, PDF, zip) without corruption

Never use string-based reads for binary files — always use []byte and avoid text-mode functions like bufio.Scanner/ReadString, which can mangle data at newline-like byte sequences.

package main

import (
    "fmt"
    "io"
    "os"
)

func copyBinaryFile(src, dst string) error {
    in, err := os.Open(src)
    if err != nil {
        return err
    }
    defer in.Close()

    out, err := os.Create(dst)
    if err != nil {
        return err
    }
    defer out.Close()

    _, err = io.Copy(out, in) // handles binary data safely
    return err
}

func main() {
    if err := copyBinaryFile("photo.jpg", "photo_copy.jpg"); err != nil {
        fmt.Println("Copy failed:", err)
        return
    }
    fmt.Println("Binary file copied successfully")
}

3. Reading fixed-size binary structures with encoding/binary

Useful for custom binary formats, protocol parsing, or reading structured records.

package main

import (
    "encoding/binary"
    "fmt"
    "os"
)

type Record struct {
    ID    uint32
    Score float64
}

func main() {
    file, err := os.Open("records.bin")
    if err != nil {
        fmt.Println(err)
        return
    }
    defer file.Close()

    var rec Record
    for {
        err := binary.Read(file, binary.LittleEndian, &rec)
        if err != nil {
            break // likely io.EOF
        }
        fmt.Printf("ID=%d Score=%.2f\n", rec.ID, rec.Score)
    }
}

To write records in the same format, use binary.Write(file, binary.LittleEndian, &rec).

4. Get file info before working with it (size, mode, mod time)

info, err := os.Stat("sample.bin")
if err != nil {
    fmt.Println(err)
    return
}
fmt.Println("Name:", info.Name())
fmt.Println("Size (bytes):", info.Size())
fmt.Println("Permissions:", info.Mode())
fmt.Println("Modified:", info.ModTime())

---

Large File Batch/Chunk Processing

When files are too large to load into memory (multi-GB logs, video files, database dumps), read and process them in fixed-size chunks using a reusable buffer.

1. Basic chunked reading

package main

import (
    "fmt"
    "io"
    "os"
)

const chunkSize = 4096 // 4KB per chunk — tune based on your use case

func main() {
    file, err := os.Open("largefile.bin")
    if err != nil {
        fmt.Println("Error opening file:", err)
        return
    }
    defer file.Close()

    buffer := make([]byte, chunkSize)
    totalBytes := 0

    for {
        bytesRead, err := file.Read(buffer)
        if bytesRead > 0 {
            // process buffer[:bytesRead] here
            totalBytes += bytesRead
        }
        if err == io.EOF {
            break
        }
        if err != nil {
            fmt.Println("Read error:", err)
            return
        }
    }

    fmt.Println("Total bytes processed:", totalBytes)
}

2. Chunked read + chunked write (streaming transform)

A common real-world pattern: read a large file in chunks, transform each chunk (e.g., encrypt, compress, hash), and write the result to another file — without ever holding the whole file in memory.

package main

import (
    "fmt"
    "io"
    "os"
)

const chunkSize = 1024 * 1024 // 1MB chunks

func processInChunks(srcPath, dstPath string) error {
    src, err := os.Open(srcPath)
    if err != nil {
        return err
    }
    defer src.Close()

    dst, err := os.Create(dstPath)
    if err != nil {
        return err
    }
    defer dst.Close()

    buffer := make([]byte, chunkSize)

    for {
        n, err := src.Read(buffer)
        if n > 0 {
            chunk := buffer[:n]

            // Example transform: uppercase-ish byte manipulation, hashing,
            // encryption, compression — replace with your own logic
            transformed := transformChunk(chunk)

            if _, werr := dst.Write(transformed); werr != nil {
                return werr
            }
        }
        if err == io.EOF {
            break
        }
        if err != nil {
            return err
        }
    }
    return nil
}

func transformChunk(data []byte) []byte {
    // Placeholder: identity transform. Swap this for real processing.
    return data
}

func main() {
    if err := processInChunks("largefile.bin", "processed_output.bin"); err != nil {
        fmt.Println("Error:", err)
        return
    }
    fmt.Println("Large file processed in chunks successfully")
}

3. Batch-processing many large files (worker pool pattern)

When you have thousands of large files to process (e.g., a folder of uploaded documents), combine chunked reading with a goroutine worker pool so multiple files are processed concurrently without exhausting memory.

package main

import (
    "fmt"
    "io"
    "os"
    "path/filepath"
    "sync"
)

const (
    chunkSize  = 512 * 1024 // 512KB
    numWorkers = 4
)

func processFile(path string) error {
    file, err := os.Open(path)
    if err != nil {
        return err
    }
    defer file.Close()

    buffer := make([]byte, chunkSize)
    for {
        n, err := file.Read(buffer)
        if n > 0 {
            // process buffer[:n]
            _ = buffer[:n]
        }
        if err == io.EOF {
            break
        }
        if err != nil {
            return err
        }
    }
    return nil
}

func worker(id int, jobs <-chan string, wg *sync.WaitGroup) {
    defer wg.Done()
    for path := range jobs {
        if err := processFile(path); err != nil {
            fmt.Printf("[worker %d] error processing %s: %v\n", id, path, err)
            continue
        }
        fmt.Printf("[worker %d] finished %s\n", id, path)
    }
}

func main() {
    dir := "large_files_folder"
    entries, err := os.ReadDir(dir)
    if err != nil {
        fmt.Println("Error reading directory:", err)
        return
    }

    jobs := make(chan string, len(entries))
    var wg sync.WaitGroup

    for i := 1; i <= numWorkers; i++ {
        wg.Add(1)
        go worker(i, jobs, &wg)
    }

    for _, entry := range entries {
        if !entry.IsDir() {
            jobs <- filepath.Join(dir, entry.Name())
        }
    }
    close(jobs)

    wg.Wait()
    fmt.Println("All files processed")
}

This pattern is a good starting point for building something like a bulk file-processing service (e.g., resizing images, scanning uploads, or transcoding — comparable in spirit to a chunked-upload/CDN pipeline).

4. Using bufio.Reader for chunked reads with buffering benefits

bufio.Reader reduces the number of actual syscalls by maintaining its own internal buffer on top of your read loop — useful when your chunk size is small and you want fewer disk reads.

package main

import (
    "bufio"
    "fmt"
    "io"
    "os"
)

func main() {
    file, err := os.Open("largefile.bin")
    if err != nil {
        fmt.Println(err)
        return
    }
    defer file.Close()

    reader := bufio.NewReaderSize(file, 64*1024) // 64KB internal buffer
    chunk := make([]byte, 8*1024)                // 8KB read chunks

    for {
        n, err := reader.Read(chunk)
        if n > 0 {
            // process chunk[:n]
        }
        if err == io.EOF {
            break
        }
        if err != nil {
            fmt.Println("Read error:", err)
            return
        }
    }
    fmt.Println("Done reading large file with buffered chunks")
}

---

Copying Files Efficiently

For most copy needs — small or large, text or binary — io.Copy is the right tool since it streams internally and never loads the full file into memory:

func copyFile(src, dst string) (int64, error) {
    in, err := os.Open(src)
    if err != nil {
        return 0, err
    }
    defer in.Close()

    out, err := os.Create(dst)
    if err != nil {
        return 0, err
    }
    defer out.Close()

    return io.Copy(out, in)
}

If you need control over the internal buffer size (e.g., for performance tuning on very large files), use io.CopyBuffer:

buf := make([]byte, 1024*1024) // 1MB buffer
written, err := io.CopyBuffer(out, in, buf)

---

Best Practices & Common Pitfalls

1. Always check and handle errors. File operations fail for many reasons (permissions, missing paths, disk full) — never ignore the error return value.
2. Always defer file.Close() right after a successful Open/Create call, so the file descriptor is released even if the function returns early.
3. Always Flush() a bufio.Writer before the program exits or the file is closed — buffered data not flushed is silently lost.
4. Don't use os.ReadFile/os.WriteFile for huge files. They load everything into memory. Use chunked reads/writes (bufio, io.Copy, manual buffers) instead.
5. Use io.Copy/io.CopyBuffer for binary data, never string-based scanning, since binary content can contain byte sequences that look like line breaks or other delimiters.
6. Pick a sensible chunk size. Too small (e.g., 512 bytes) causes excessive syscalls; too large wastes memory. 32KB–1MB is a common sweet spot, adjust based on profiling.
7. Use os.O_EXCL with os.O_CREATE if you need atomic "create only if it doesn't exist" semantics (avoids race conditions in concurrent environments).
8. Watch permissions. 0644 is common for regular files, 0600 if only the owner should read/write (e.g., secrets, private keys).
9. Use errors.Is(err, os.ErrNotExist) (or os.ErrExist) instead of comparing error strings — it's the idiomatic, version-safe way to check specific file errors.
10. For concurrent batch processing, bound your worker pool. Opening thousands of files simultaneously can exhaust OS file descriptor limits — a worker pool with a fixed number of goroutines avoids this.

---

Quick Reference Table

Task	Recommended Function(s)
Read whole small file	os.ReadFile
Read line by line	bufio.NewScanner
Read with custom control	bufio.NewReader + ReadString/Read
Write whole file (overwrite)	os.WriteFile
Write with buffering	os.Create + bufio.NewWriter
Append to a file	os.OpenFile with O_APPEND|O_CREATE|O_WRONLY
Delete a file	os.Remove
Delete a folder recursively	os.RemoveAll
Binary read/write	os.ReadFile/os.WriteFile, encoding/binary
Copy any file (safe for binary)	io.Copy / io.CopyBuffer
Process large file in chunks	Manual buffer loop with file.Read(buffer)
Batch-process many large files	Worker pool (goroutines + channels) + chunked reads

---

With these patterns you can handle everything from a tiny config file to gigabyte-scale batch pipelines — reading, writing, appending, deleting, and safely handling binary data — all using Go's standard library, no third-party dependencies required.