mb_convert_encoding

For my last project I needed to convert several CSV files from Windows-1250 to UTF-8, and after several days of searching around I found a function that is partially solved my problem, but it still has not transformed all the characters. So I made ​​this:function w1250_to_utf8($text) {    // map based on:    // http://konfiguracja.c0.pl/iso02vscp1250en.html    // http://konfiguracja.c0.pl/webpl/index_en.html#examp    // http://www.htmlentities.com/html/entities/    $map = array(        chr(0x8A) => chr(0xA9),        chr(0x8C) => chr(0xA6),        chr(0x8D) => chr(0xAB),        chr(0x8E) => chr(0xAE),        chr(0x8F) => chr(0xAC),        chr(0x9C) => chr(0xB6),        chr(0x9D) => chr(0xBB),        chr(0xA1) => chr(0xB7),        chr(0xA5) => chr(0xA1),        chr(0xBC) => chr(0xA5),        chr(0x9F) => chr(0xBC),        chr(0xB9) => chr(0xB1),        chr(0x9A) => chr(0xB9),        chr(0xBE) => chr(0xB5),        chr(0x9E) => chr(0xBE),        chr(0x80) => '€',        chr(0x82) => '‚',        chr(0x84) => '„',        chr(0x85) => '…',        chr(0x86) => '†',        chr(0x87) => '‡',        chr(0x89) => '‰',        chr(0x8B) => '‹',        chr(0x91) => '‘',        chr(0x92) => '’',        chr(0x93) => '“',        chr(0x94) => '”',        chr(0x95) => '•',        chr(0x96) => '–',        chr(0x97) => '—',        chr(0x99) => '™',        chr(0x9B) => '’',        chr(0xA6) => '¦',        chr(0xA9) => '©',        chr(0xAB) => '«',        chr(0xAE) => '®',        chr(0xB1) => '±',        chr(0xB5) => 'µ',        chr(0xB6) => '¶',        chr(0xB7) => '·',        chr(0xBB) => '»',    );    return html_entity_decode(mb_convert_encoding(strtr($text, $map), 'UTF-8', 'ISO-8859-2'), ENT_QUOTES, 'UTF-8');}

I've been trying to find the charset of a norwegian (with a lot of ø, æ, å) txt file written on a Mac, i've found it in this way:

<?php
$text = "A strange string to pass, maybe with some ø, æ, å characters.";

foreach(mb_list_encodings() as $chr){
        echo mb_convert_encoding($text, 'UTF-8', $chr)." : ".$chr."<br>";    
 } 
?>

The line that looks good, gives you the encoding it was written in.

Hope can help someone

Hey guys. For everybody who's looking for a function that is converting an iso-string to utf8 or an utf8-string to iso, here's your solution:public function encodeToUtf8($string) {     return mb_convert_encoding($string, "UTF-8", mb_detect_encoding($string, "UTF-8, ISO-8859-1, ISO-8859-15", true));}public function encodeToIso($string) {     return mb_convert_encoding($string, "ISO-8859-1", mb_detect_encoding($string, "UTF-8, ISO-8859-1, ISO-8859-15", true));}For me these functions are working fine. Give it a try

If you have what looks like ISO-8859-1, but it includes "smart quotes" courtesy of Microsoft software, or people cutting and pasting content from Microsoft software, then what you're actually dealing with is probably Windows-1252. Try this:<?php$cleanText = mb_convert_encoding($text, 'UTF-8', 'Windows-1252');?>The annoying part is that the auto detection (ie: the mb_detect_encoding function) will often think Windows-1252 is ISO-8859-1. Close, but no cigar. This is critical if you're then trying to do unserialize on the resulting text, because the byte count of the string needs to be perfect.

Text-encoding HTML-ENTITIES will be deprecated as of PHP 8.2.To convert all non-ASCII characters into entities (to produce pure 7-bit HTML output), I was using:<?phpecho mb_convert_encoding( htmlspecialchars( $text, ENT_QUOTES, 'UTF-8' ), 'HTML-ENTITIES', 'UTF-8' );?>I can get the identical result with:<?phpecho mb_encode_numericentity( htmlentities( $text, ENT_QUOTES, 'UTF-8' ), [0x80, 0x10FFFF, 0, ~0], 'UTF-8' );?>The output contains well-known named entities for some often used characters and numeric entities for the rest.

aaron, to discard unsupported characters instead of printing a ?, you might as well simply set the configuration directive:mbstring.substitute_character = "none"in your php.ini. Be sure to include the quotes around none. Or at run-time with<?phpini_set('mbstring.substitute_character', "none");?>

many people below talk about using 
<?php
    mb_convert_encode($s,'HTML-ENTITIES','UTF-8');
?>
to convert non-ascii code into html-readable stuff.  Due to my webserver being out of my control, I was unable to set the database character set, and whenever PHP made a copy of my $s variable that it had pulled out of the database, it would convert it to nasty latin1 automatically and not leave it in it's beautiful UTF-8 glory.

So [insert korean characters here] turned into ?????.

I found myself needing to pass by reference (which of course is deprecated/nonexistent in recent versions of PHP)
so instead of
<?php
    mb_convert_encode(&$s,'HTML-ENTITIES','UTF-8');
?>
which worked perfectly until I upgraded, so I had to use
<?php
    call_user_func_array('mb_convert_encoding', array(&$s,'HTML-ENTITIES','UTF-8'));
?>

Hope it helps someone else out

My solution below was slightly incorrect, so here is the correct version (I posted at the end of a long day, never a good idea!)Again, this is a quick and dirty solution to stop mb_convert_encoding from filling your string with question marks whenever it encounters an illegal character for the target encoding. <?phpfunction convert_to ( $source, $target_encoding )    {    // detect the character encoding of the incoming file    $encoding = mb_detect_encoding( $source, "auto" );           // escape all of the question marks so we can remove artifacts from    // the unicode conversion process    $target = str_replace( "?", "[question_mark]", $source );           // convert the string to the target encoding    $target = mb_convert_encoding( $target, $target_encoding, $encoding);           // remove any question marks that have been introduced because of illegal characters    $target = str_replace( "?", "", $target );           // replace the token string "[question_mark]" with the symbol "?"    $target = str_replace( "[question_mark]", "?", $target );       return $target;    }?>Hope this helps someone! (Admins should feel free to delete my previous, incorrect, post for clarity)-A

If you are trying to generate a CSV (with extended chars) to be opened at Exel for Mac, the only that worked for me was:
<?php mb_convert_encoding( $CSV, 'Windows-1252', 'UTF-8'); ?>

I also tried this:

<?php
//Separado OK, chars MAL
iconv('MACINTOSH', 'UTF8', $CSV);
//Separado MAL, chars OK
chr(255).chr(254).mb_convert_encoding( $CSV, 'UCS-2LE', 'UTF-8');
?>

But the first one didn't show extended chars correctly, and the second one, did't separe fields correctly

To add to the Flash conversion comment below, here's how I convert back from what I've stored in a database after converting from Flash HTML text field output, in order to load it back into a Flash HTML text field:function htmltoflash($htmlstr){  return str_replace("&lt;br /&gt;","\n",    str_replace("<","&lt;",      str_replace(">","&gt;",        mb_convert_encoding(html_entity_decode($htmlstr),        "UTF-8","ISO-8859-1"))));}

Note that `mb_convert_encoding($val, 'HTML-ENTITIES')` does not escape '\'', '"', '<', '>', or '&'.

It appears that when dealing with an unknown "from encoding" the function will both throw an E_WARNING and proceed to convert the string from ISO-8859-1 to the "to encoding".

When you need to convert from HTML-ENTITIES, but your UTF-8 string is partially broken (not all chars in UTF-8) - in this case passing string to mb_convert_encoding($string, 'UTF-8', 'HTML-ENTITIES'); - corrupts chars in string even more. In this case you need to replace html entities gradually to preserve character good encoding. I wrote such closure for this job :<?php$decode_entities = function($string) {        preg_match_all("/&#?\w+;/", $string, $entities, PREG_SET_ORDER);        $entities = array_unique(array_column($entities, 0));        foreach ($entities as $entity) {            $decoded = mb_convert_encoding($entity, 'UTF-8', 'HTML-ENTITIES');            $string = str_replace($entity, $decoded, $string);        }        return $string;    };?>

/** * Convert Windows-1250 to UTF-8 * Based on https://www.php.net/manual/en/function.mb-convert-encoding.php#112547 */class TextConverter{    private const ENCODING_TO = 'UTF-8';    private const ENCODING_FROM = 'ISO-8859-2';    private array $mapChrChr = [        0x8A => 0xA9,        0x8C => 0xA6,        0x8D => 0xAB,        0x8E => 0xAE,        0x8F => 0xAC,        0x9C => 0xB6,        0x9D => 0xBB,        0xA1 => 0xB7,        0xA5 => 0xA1,        0xBC => 0xA5,        0x9F => 0xBC,        0xB9 => 0xB1,        0x9A => 0xB9,        0xBE => 0xB5,        0x9E => 0xBE    ];    private array $mapChrString = [        0x80 => '€',        0x82 => '‚',        0x84 => '„',        0x85 => '…',        0x86 => '†',        0x87 => '‡',        0x89 => '‰',        0x8B => '‹',        0x91 => '‘',        0x92 => '’',        0x93 => '“',        0x94 => '”',        0x95 => '•',        0x96 => '–',        0x97 => '—',        0x99 => '™',        0x9B => '’',        0xA6 => '¦',        0xA9 => '©',        0xAB => '«',        0xAE => '®',        0xB1 => '±',        0xB5 => 'µ',        0xB6 => '¶',        0xB7 => '·',        0xBB => '»'    ];    /**     * @param $text     * @return string     */    public function execute($text): string    {        $map = $this->prepareMap();        return html_entity_decode(            mb_convert_encoding(strtr($text, $map), self::ENCODING_TO, self::ENCODING_FROM),            ENT_QUOTES,            self::ENCODING_TO        );    }    /**     * @return array     */    private function prepareMap(): array    {        $maps[] = $this->arrayMapAssoc(function ($k, $v) {            return [chr($k), chr($v)];        }, $this->mapChrChr);        $maps[] = $this->arrayMapAssoc(function ($k, $v) {            return [chr($k), $v];        }, $this->mapChrString);        return array_merge([], ...$maps);    }    /**     * @param callable $function     * @param array $array     * @return array     */    private function arrayMapAssoc(callable $function, array $array): array    {        return array_column(            array_map(                $function,                array_keys($array),                $array            ),            1,            0        );    }}

instead of ini_set(), you can try thismb_substitute_character("none");

Clean a string for use as filename by simply replacing all unwanted characters with underscore (ASCII converts to 7bit). It removes slightly more chars than necessary. Hope its useful. $fileName = 'Test:!"$%&/()=ÖÄÜöäü<<';echo strtr(mb_convert_encoding($fileName,'ASCII'),     ' ,;:?*#!§$%&/(){}<>=`´|\\\'"',     '____________________________');

Another sample of recoding without MultiByte enabling.(Russian koi->win, if input in win-encoding already, function recode() returns unchanged string)<?php  // 0 - win  // 1 - koi  function detect_encoding($str) {    $win = 0;    $koi = 0;    for($i=0; $i<strlen($str); $i++) {      if( ord($str[$i]) >224 && ord($str[$i]) < 255) $win++;      if( ord($str[$i]) >192 && ord($str[$i]) < 223) $koi++;    }    if( $win < $koi ) {      return 1;    } else return 0;  }  // recodes koi to win  function koi_to_win($string) {    $kw = array(128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183,  184, 185, 186, 187, 188, 189, 190, 191, 254, 224, 225, 246, 228, 229, 244, 227, 245, 232, 233, 234, 235, 236, 237, 238, 239, 255, 240, 241, 242, 243, 230, 226, 252, 251, 231, 248, 253, 249, 247, 250, 222, 192, 193, 214, 196, 197, 212, 195, 213, 200, 201, 202, 203, 204, 205, 206, 207, 223, 208, 209, 210, 211, 198, 194, 220, 219, 199, 216, 221, 217, 215, 218);    $wk = array(128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183,  184, 185, 186, 187, 188, 189, 190, 191, 225, 226, 247, 231, 228, 229, 246, 250, 233, 234, 235, 236, 237, 238, 239, 240, 242,  243, 244, 245, 230, 232, 227, 254, 251, 253, 255, 249, 248, 252, 224, 241, 193, 194, 215, 199, 196, 197, 214, 218, 201, 202, 203, 204, 205, 206, 207, 208, 210, 211, 212, 213, 198, 200, 195, 222, 219, 221, 223, 217, 216, 220, 192, 209);    $end = strlen($string);    $pos = 0;    do {      $c = ord($string[$pos]);      if ($c>128) {        $string[$pos] = chr($kw[$c-128]);      }    } while (++$pos < $end);    return $string;  }  function recode($str) {    $enc = detect_encoding($str);    if ($enc==1) {      $str = koi_to_win($str);    }    return $str;  }?>

// convert UTF8 to DOS = CP850 //// $utf8_text=UTF8-Formatted text;// $dos=CP850-Formatted text;// have fun$dos = mb_convert_encoding($utf8_text, "CP850", mb_detect_encoding($utf8_text, "UTF-8, CP850, ISO-8859-15", true));

Why did you use the php html encode functions? mbstring has it's own Encoding which is (as far as I tested it) much more usefull:HTML-ENTITIESExample:$text = mb_convert_encoding($text, 'HTML-ENTITIES', "UTF-8");

If you are attempting to convert "UTF-8" text to "ISO-8859-1" and the result is always returning in "ASCII", place the following line of code before the mb_convert_encoding:mb_detect_order(array('UTF-8', 'ISO-8859-1'));It is necessary to force a specific search order for the conversion to work

For those wanting to convert from $set to MacRoman, use iconv():<?php$string = iconv('UTF-8', 'macintosh', $string);?>('macintosh' is the IANA name for the MacRoman character set.)

As an alternative to Johannes's suggestion for converting strings from other character sets to a 7bit representation while not just deleting latin diacritics, you might try this:<?php$text = iconv($from_enc, 'US-ASCII//TRANSLIT', $text);?>The only disadvantage is that it does not convert "ä" to "ae", but it handles punctuation and other special characters better.-- David

When converting Japanese strings to ISO-2022-JP or JIS on PHP >= 5.2.1, you can use "ISO-2022-JP-MS" instead of them.Kishu-Izon (platform dependent) characters are converted correctly with the encoding, as same as with eucJP-win or with SJIS-win.

be careful when converting from iso-8859-1 to utf-8.even if you explicitly specify the character encoding of a page as iso-8859-1(via headers and strict xml defs), windows 2000 will ignore that and interpret it as whatever character set it has natively installed. for example, i wrote char #128 into a page, with char encoding iso-8859-1, and it displayed in internet explorer (& mozilla) as a euro symbol.it should have displayed a box, denoting that char #128 is undefined in iso-8859-1. The problem was it was displaying in "Windows: western europe" (my native character set).this led to confusion when i tried to convert this euro to UTF-8 via mb_convert_encoding()  IE displays UTF-8 correctly- and because PHP correctly converted #128 into a box in UTF-8, IE would show a box.so all i saw was mb_convert_encoding() converting a euro symbol into a box. It took me a long time to figure out what was going on.

rodrigo at bb2 dot co dot jp wrote that inconv works better than mb_convert_encoding, I find that when converting from uft8 to shift_jis $conv_str = mb_convert_encoding($str,$toCS,$fromCS); works while$conv_str = iconv($fromCS,$toCS.'//IGNORE',$str); removes tildes from $str.

If you want to convert japanese to ISO-2022-JP it is highly recommended to use ISO-2022-JP-MS as the target encoding instead. This includes the extended character set and avoids ? in the text. For example the often used "1 in a circle" ① will be correctly converted then.

For the php-noobs (like me) - working with flash and php.Here's a simple snippet of code that worked great for me, getting php to show special Danish characters, from a Flash email form:<?php// Name Escape$escName = mb_convert_encoding($_POST["Name"], "ISO-8859-1", "UTF-8");// message escape$escMessage = mb_convert_encoding($_POST["Message"], "ISO-8859-1", "UTF-8");// Headers.. and so on...?>

For those who can´t use mb_convert_encoding() to convert from one charset to another as a metter of lower version of php, try iconv().I had this problem converting to japanese charset:$txt=mb_convert_encoding($txt,'SJIS',$this->encode);And I could fix it by using this:$txt = iconv('UTF-8', 'SJIS', $txt);Maybe it´s helpfull for someone else! ;)

I\'d like to share some code to convert latin diacritics to their
traditional 7bit representation, like, for example,

- à,ç,é,î,... to a,c,e,i,...
- ß to ss
- ä,Ä,... to ae,Ae,...
- ë,... to e,...

(mb_convert \"7bit\" would simply delete any offending characters). 

I might have missed on your country\'s typographic 
conventions--correct me then. 
<?php
/**
 * @args string $text line of encoded text
 *       string $from_enc (encoding type of $text, e.g. UTF-8, ISO-8859-1)
 *
 * @returns 7bit representation
 */
function to7bit($text,$from_enc) {
    $text = mb_convert_encoding($text,\'HTML-ENTITIES\',$from_enc);
    $text = preg_replace(
        array(\'/ß/\',\'/&(..)lig;/\',
             \'/&([aouAOU])uml;/\',\'/&(.)[^;]*;/\'),
        array(\'ss\',\"$1\",\"$1\".\'e\',\"$1\"),
        $text);
    return $text;
}   
?>

Enjoy :-)
Johannes

==
[EDIT BY danbrown AT php DOT net: Author provided the following update on 27-FEB-2012.]
==

An addendum to my "to7bit" function referenced below in the notes. 
The function is supposed to solve the problem that some languages require a different 7bit rendering of special (umlauted) characters for sorting or other applications. For example, the German ß ligature is usually written "ss" in 7bit context. Dutch ÿ is typically rendered "ij" (not "y"). 

The original function works well with word (alphabet) character entities and I've seen it used in many places. But non-word entities cause funny results:
E.g., "©" is rendered as "c", "­" as "s" and "&rquo;" as "r". 
The following version fixes this by converting non-alphanumeric characters (also chains thereof) to '_'.

<?php
/**
 * @args string $text line of encoded text
 *       string $from_enc (encoding type of $text, e.g. UTF-8, ISO-8859-1)
 *
 * @returns 7bit representation
 */
function to7bit($text,$from_enc) {
    $text = preg_replace(/W+/,'_',$text);
    $text = mb_convert_encoding($text,'HTML-ENTITIES',$from_enc);
    $text = preg_replace(
        array('/ß/','/&(..)lig;/',
             '/&([aouAOU])uml;/','/ÿ/','/&(.)[^;]*;/'),
        array('ss',"$1","$1".'e','ij',"$1"),
        $text);
    return $text;
}  
?>

Enjoy again,
Johannes

Here's a tip for anyone using Flash and PHP for storing HTML output submitted from a Flash text field in a database or whatever.Flash submits its HTML special characters in UTF-8, so you can use the following function to convert those into HTML entity characters:function utf8html($utf8str){  return htmlentities(mb_convert_encoding($utf8str,"ISO-8859-1","UTF-8"));}