How to Tokenize a String in C++

  1. Use the find and substr Functions to Tokenize a String in C++
  2. Use the std::stringstream and getline Functions to Tokenize a String in C++
  3. Use the istringstream and copy Algorithm to Tokenize a String in C++

This article will explain several methods of how to tokenize a string in C++.

Use the find and substr Functions to Tokenize a String in C++

std::string class has a built-in find function to search for a sequence of characters in a given string object. The find function returns the first character’s position found in the string and returns npos if not found. The find function call is inserted in the if statement to iterate over string until the last string token is extracted.

Notice that a user can specify any string type delimiter and pass it to the find method. The tokens are pushed to the vector of strings and already processed part removed with the erase() function on each iteration.

#include <iostream>
#include <string>
#include <iterator>
#include <vector>
#include <algorithm>

using std::cout; using std::cin;
using std::endl; using std::string;
using std::vector;

string text = "Think you are escaping and run into yourself.";
int main(){
    string delim = " ";
    vector<string> words{};

    size_t pos = 0;
    while ((pos = text.find(delim)) != string::npos) {
        words.push_back(text.substr(0, pos));
        text.erase(0, pos + delim.length());
    }
    if (!text.empty())
        words.push_back(text.substr(0, pos));
    for (const auto &str : words) {
        cout << str << endl;
    }

    return EXIT_SUCCESS;
}

Output:

Think
you
are
escaping
and
run
into
yourself.

Use the std::stringstream and getline Functions to Tokenize a String in C++

stringstream can be utilized to ingest a string to be processed and use getline to extract tokens until the given delimiter is found. Note that this method only works with single-character delimiters.

#include <iostream>
#include <string>
#include <sstream>
#include <iterator>
#include <vector>
#include <algorithm>

using std::cout; using std::cin;
using std::endl; using std::string;
using std::vector; using std::istringstream;
using std::stringstream;

int main(){
    string text = "Think you are escaping and run into yourself.";
    char del = ' ';
    vector<string> words{};

    stringstream sstream(text);
    string word;
    while (std::getline(sstream, word, del))
       words.push_back(word);

    for (const auto &str : words) {
       cout << str << endl;
    }

    return EXIT_SUCCESS;
}

Output:

Think
you
are
escaping
and
run
into
yourself.

Use the istringstream and copy Algorithm to Tokenize a String in C++

Alternatively, we can use the copy function from the <algorithm> header and extract string tokens on white space delimiters. In the following example, we only iterate and stream tokens to the standard output. To process the string with the copy method, we insert it in istringstream and utilize its iterators.

#include <iostream>
#include <string>
#include <sstream>
#include <iterator>
#include <vector>
#include <algorithm>

using std::cout; using std::cin;
using std::endl; using std::string;
using std::vector; using std::istringstream;
using std::stringstream;

int main(){
    string text = "Think you are escaping and run into yourself.";
    string delim = " ";
    vector<string> words{};

    istringstream iss(text);
    copy(std::istream_iterator<string>(iss),
        std::istream_iterator<string>(),
        std::ostream_iterator<string>(cout, "\n"));

    return EXIT_SUCCESS;
}

Output:

Think
you
are
escaping
and
run
into
yourself.

Related Article - C++ String

  • How to Compare Two Strings Ignoring the Case in C++
  • How to Print a String in C++