Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Welcome To Ask or Share your Answers For Others

Categories

0 votes
190 views
in Technique[技术] by (71.8m points)

python - Find emojis in a tweet as whole clusters and not as individual chars

So first question I've ever asked on here, and its about emojis. I'm sorry.

I am making a twitter bot in python with the help of Tweepy, and regex ( also tried python-pcre ) that will analyse a tweet of a given user, and record the number of times a word or emoji was used. I can do most of this just fine. My problems start with the emojis.

I was under the impression that when using X, (in both regex and python-pcre) will find the eXtended grapheme clusters. Not just the individual ones. I read in another post What does the expression X match when inside a RegEx? that X follows a set of guidelines to determine if the next char should be clustered, but will always return at least 1.

I tried the first and second solution over at this post: How to extract all the emojis from text?.

The first one acted as expected. Grabs individual code-points and adds them to a list. Perfect for single code-point emojis, but I need to capture emojis with multiple code-points, and single code-point emojis.

The second solution one is where I am having problems. According to the post this function should print the emojis in a string, in clusters, separated by spaces.

def split_count(self, text):
        emoji_list = []
        data = regex.findall(r'X', text)
        for word in data:
            if any(char in emoji.UNICODE_EMOJI for char in word):
                emoji_list.append(word)
        return emoji_list

When called like:

counter = self.split_count(tweet)
print(' '.join(emoji for emoji in counter))

Should result in:

?? ???????????

However when I run it I get:

?? ?? ?? ?? ??

100% not clustered.

Why is this happening? This has been bugging me for a couple days now.

See Question&Answers more detail:os

与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
Welcome To Ask or Share your Answers For Others

1 Reply

0 votes
by (71.8m points)

Note that X matches a single code point, but emojis may contain more than one, you yourself mention it in the question.

You should use an emoji parsing library that you are already using to get the emojis out of the text.

Use re, no need for regex:

import re
from emoji import UNICODE_EMOJI

e_list = UNICODE_EMOJI.keys()
extract_emojis = re.compile("|".join(map(re.escape, sorted(e_list,key=len,reverse=True))))

#...

def split_count(self, text):
        return extract_emojis.findall(text)
        

Note that with this code, running it against a file containing all unique occurrences of emojis from Emoji List, v13.1, I cannot get any results due to timeout.

So, to match those latest emojis you need a custom regex:

emoji_rx = r"[#*0-9]uFE0F?u20E3|?uFE0F?|[?u203Cu2049u2122u2139u2194-u2199u21A9u21AA]uFE0F?|[u231Au231B]|[u2328u23CF]uFE0F?|[u23E9-u23EC]|[u23ED-u23EF]uFE0F?|u23F0|[u23F1u23F2]uFE0F?|u23F3|[u23F8-u23FAu24C2u25AAu25ABu25B6u25C0u25FBu25FC]uFE0F?|[u25FDu25FE]|[u2600-u2604u260Eu2611]uFE0F?|[u2614u2615]|u2618uFE0F?|u261D[uFE0FU0001F3FB-U0001F3FF]?|[u2620u2622u2623u2626u262Au262Eu262Fu2638-u263Au2640u2642]uFE0F?|[u2648-u2653]|[u265Fu2660u2663u2665u2666u2668u267Bu267E]uFE0F?|u267F|u2692uFE0F?|u2693|[u2694-u2697u2699u269Bu269Cu26A0]uFE0F?|u26A1|u26A7uFE0F?|[u26AAu26AB]|[u26B0u26B1]uFE0F?|[u26BDu26BEu26C4u26C5]|u26C8uFE0F?|u26CE|[u26CFu26D1u26D3]uFE0F?|u26D4|u26E9uFE0F?|u26EA|[u26F0u26F1]uFE0F?|[u26F2u26F3]|u26F4uFE0F?|u26F5|[u26F7u26F8]uFE0F?|u26F9(?:u200D[u2640u2642]uFE0F?|[uFE0FU0001F3FB-U0001F3FF](?:u200D[u2640u2642]uFE0F?)?)?|[u26FAu26FD]|u2702uFE0F?|u2705|[u2708u2709]uFE0F?|[u270Au270B][U0001F3FB-U0001F3FF]?|[u270Cu270D][uFE0FU0001F3FB-U0001F3FF]?|u270FuFE0F?|[u2712u2714u2716u271Du2721]uFE0F?|u2728|[u2733u2734u2744u2747]uFE0F?|[u274Cu274Eu2753-u2755u2757]|u2763uFE0F?|u2764(?:u200D[U0001F525U0001FA79]|uFE0F(?:u200D[U0001F525U0001FA79])?)?|[u2795-u2797]|u27A1uFE0F?|[u27B0u27BF]|[u2934u2935u2B05-u2B07]uFE0F?|[u2B1Bu2B1Cu2B50u2B55]|[u3030u303Du3297u3299]uFE0F?|[U0001F004U0001F0CF]|[U0001F170U0001F171U0001F17EU0001F17F]uFE0F?|[U0001F18EU0001F191-U0001F19A]|U0001F1E6[U0001F1E8-U0001F1ECU0001F1EEU0001F1F1U0001F1F2U0001F1F4U0001F1F6-U0001F1FAU0001F1FCU0001F1FDU0001F1FF]|U0001F1E7[U0001F1E6U0001F1E7U0001F1E9-U0001F1EFU0001F1F1-U0001F1F4U0001F1F6-U0001F1F9U0001F1FBU0001F1FCU0001F1FEU0001F1FF]|U0001F1E8[U0001F1E6U0001F1E8U0001F1E9U0001F1EB-U0001F1EEU0001F1F0-U0001F1F5U0001F1F7U0001F1FA-U0001F1FF]|U0001F1E9[U0001F1EAU0001F1ECU0001F1EFU0001F1F0U0001F1F2U0001F1F4U0001F1FF]|U0001F1EA[U0001F1E6U0001F1E8U0001F1EAU0001F1ECU0001F1EDU0001F1F7-U0001F1FA]|U0001F1EB[U0001F1EE-U0001F1F0U0001F1F2U0001F1F4U0001F1F7]|U0001F1EC[U0001F1E6U0001F1E7U0001F1E9-U0001F1EEU0001F1F1-U0001F1F3U0001F1F5-U0001F1FAU0001F1FCU0001F1FE]|U0001F1ED[U0001F1F0U0001F1F2U0001F1F3U0001F1F7U0001F1F9U0001F1FA]|U0001F1EE[U0001F1E8-U0001F1EAU0001F1F1-U0001F1F4U0001F1F6-U0001F1F9]|U0001F1EF[U0001F1EAU0001F1F2U0001F1F4U0001F1F5]|U0001F1F0[U0001F1EAU0001F1EC-U0001F1EEU0001F1F2U0001F1F3U0001F1F5U0001F1F7U0001F1FCU0001F1FEU0001F1FF]|U0001F1F1[U0001F1E6-U0001F1E8U0001F1EEU0001F1F0U0001F1F7-U0001F1FBU0001F1FE]|U0001F1F2[U0001F1E6U0001F1E8-U0001F1EDU0001F1F0-U0001F1FF]|U0001F1F3[U0001F1E6U0001F1E8U0001F1EA-U0001F1ECU0001F1EEU0001F1F1U0001F1F4U0001F1F5U0001F1F7U0001F1FAU0001F1FF]|U0001F1F4U0001F1F2|U0001F1F5[U0001F1E6U0001F1EA-U0001F1EDU0001F1F0-U0001F1F3U0001F1F7-U0001F1F9U0001F1FCU0001F1FE]|U0001F1F6U0001F1E6|U0001F1F7[U0001F1EAU0001F1F4U0001F1F8U0001F1FAU0001F1FC]|U0001F1F8[U0001F1E6-U0001F1EAU0001F1EC-U0001F1F4U0001F1F7-U0001F1F9U0001F1FBU0001F1FD-U0001F1FF]|U0001F1F9[U0001F1E6U0001F1E8U0001F1E9U0001F1EB-U0001F1EDU0001F1EF-U0001F1F4U0001F1F7U0001F1F9U0001F1FBU0001F1FCU0001F1FF]|U0001F1FA[U0001F1E6U0001F1ECU0001F1F2U0001F1F3U0001F1F8U0001F1FEU0001F1FF]|U0001F1FB[U0001F1E6U0001F1E8U0001F1EAU0001F1ECU0001F1EEU0001F1F3U0001F1FA]|U0001F1FC[U0001F1EBU0001F1F8]|U0001F1FDU0001F1F0|U0001F1FE[U0001F1EAU0001F1F9]|U0001F1FF[U0001F1E6U0001F1F2U0001F1FC]|U0001F201|U0001F202uFE0F?|[U0001F21AU0001F22FU0001F232-U0001F236]|U0001F237uFE0F?|[U0001F238-U0001F23AU0001F250U0001F251U0001F300-U0001F320]|[U0001F321U0001F324-U0001F32C]uFE0F?|[U0001F32D-U0001F335]|U0001F336uFE0F?|[U0001F337-U0001F37C]|U0001F37DuFE0F?|[U0001F37E-U0001F384]|U0001F385[U0001F3FB-U0001F3FF]?|[U0001F386-U0001F393]|[U0001F396U0001F397U0001F399-U0001F39BU0001F39EU0001F39F]uFE0F?|[U0001F3A0-U0001F3C1]|U0001F3C2[U0001F3FB-U0001F3FF]?|[U0001F3C3U0001F3C4](?:u200D[u2640u2642]uFE0F?|[U0001F3FB-U0001F3FF](?:u200D[u2640u2642]uFE0F?)?)?|[U0001F3C5U0001F3C6]|U0001F3C7[U0001F3FB-U0001F3FF]?|[U0001F3C8U0001F3C9]|U0001F3CA(?:u200D[u2640u2642]uFE0F?|[U0001F3FB-U0001F3FF](?:u200D[u2640u2642]uFE0F?)?)?|[U0001F3CBU0001F3CC](?:u200D[u2640u2642]uFE0F?|[uFE0FU0001F3FB-U0001F3FF](?:u200D[u2640u2642]uFE0F?)?)?|[U0001F3CDU0001F3CE]uFE0F?|[U0001F3CF-U0001F3D3]|[U0001F3D4-U0001F3DF]uFE0F?|[U0001F3E0-U0001F3F0]|U0001F3F3(?:u200D(?:u26A7uFE0F?|U0001F308)|uFE0F(?:u200D(?:u26A7uFE0F?|U0001F308))?)?|U0001F3F4(?:u200Du2620uFE0F?|U000E0067U000E0062(?:U000E0065U000E006EU000E0067|U000E0073U000E0063U000E0074|U000E0077U000E006CU000E0073)U000E007F)?|[U0001F3F5U0001F3F7]uFE0F?|[U0001F3F8-U0001F407]|U0001F408(?:u200Du2B1B)?|[U0001F409-U0001F414]|U0001F415(?:u200DU0001F9BA)?|[U0001F416-U0001F43A]|U0001F43B(?:u200Du2744uFE0F?)?|[U0001F43C-U0001F43E]|U0001F43FuFE0F?|U0001F440|U0001F441(?:u200DU0001F5E8uFE0F?|uFE0F(?:u200DU0001F5E8uFE0F?)?)?|[U0001F442U0001F443][U0001F3FB-U0001F3FF]?|[U0001F444U0001F445]|[U0001F446-U0001F450][U0001F3FB-U0001F3FF]?|[U0001F451-U0001F465]|[U0001F466U0001F467][U0001F3FB-U0001F3FF]?|U0001F468(?:u200D(?:[u2695u2696u2708]uFE0F?|u2764uFE0F?u200D(?:U0001F48Bu200D)?U0001F468|[U0001F33EU0001F373U0001F37CU0001F393U0001F3A4U0001F3A8U0001F3EBU0001F3ED]|U0001F466(?:u200DU0001F466)?|U0001F467(?:u200D[U0001F466U0001F467])?|[U0001F468U0001F469]u200D(?:U0001F466(?:u200DU0001F466)?|U0001F467(?:u200D[U0001F466U0001F467])?)|[U0001F4BBU0001F4BCU0001F527U0001F52CU0001F680U0001F692U0001F9AF-U0001F9B3U0001F9BCU0001F9BD])|U0001F3FB(?:u200D(?:[u2695u2696u2708]uFE0F?|u2764uFE0F?u200D(?:U0001F48Bu200D)?U0001F468[U0001F3FB-U0001F3FF]|[U0001F33EU0001F373U0001F37CU0001F393U0001F3A4U0001F3A8U0001F3EBU0001F3EDU0001F4BBU0001F4BCU0001F527U0001F52CU0001F680U0001F692]|U0001F91Du200DU0001F468[U0001F3FC-U0001F3FF]|[U0001F9AF-U0001F9B3U0001F9BCU0001F9BD]))?|U0001F3FC(?:u200D(?:[u2695u2696u2708]uFE0F?|u2764uFE0F?u200D(?:U0001F48Bu200D)?U0001F468[U0001F3FB-U0001F3FF]|[U0001F33EU0001F373U0001F37CU0001F393U0001F3A4U0001F3A8U0001F3EBU0001F3EDU0001F4BBU0001F4BCU0001F527U0001F52CU0001F680U0001F692]|U0001F91Du200DU0001F468[U0001F3FBU0001F3FD-U0001F3FF]|[U0001F9AF-U0001F9B3U0001F9BCU0001F9BD]))?|U0001F3FD(?:u200D(?:[u2695u2696u2708]uFE0F?|u2764uFE0F?u200D(?:U0001F48Bu200D)?U0001F468[U0001F3FB-U0001F3FF]|[U0001F33EU0001F373U0001F37CU0001F393U0001F3A4U0001F3A8U0001F3EBU0001F3EDU0001F4BBU0001F4BCU0001F527U0001F52CU0001F680U0001F692]|U0001F91Du200DU0001F468[U0001F3FBU0001F3FCU0001F3FEU0001F3FF]|[U0001F9AF-U0001F9B3U0001F9BCU0001F9BD]))?|U0001F3FE(?:u200D(?:[u2695u2696u2708]uFE0F?|u2764uFE0F?u200D(?:U0001F48Bu200D)?U0001F468[U0001F3FB-U0001F3FF]|[U0001F33EU0001F373U0001F37CU0001F393U0001F3A4U0001F3A8U0001F3EBU0001F3EDU0001F4BBU0001F4BCU0001F527U0001F52CU0001F680U0001F692]|U0001F91Du200DU0001F468[U0001F3FB-U0001F3FDU0001F3FF]|[U0001F9AF-U0001F9B3U0001F9BCU0001F9BD]))?|U0001F3FF(?:u200D(?:[u2695u2696u2708]uFE0F?|u2764uFE0F?u200D(?:U0001F48Bu200D)?U0001F468[U0001F3FB-U0001F3FF]|[U0001F33EU0001F373U0001F37CU0001F393U0001F3A4U0001F3A8U0001F3EBU0001F3EDU0001F4BBU0001F4BCU0001F527U0001F52CU0001F680U0001F692]|U0001F91Du200DU0001F468[U0001F3FB-U0001F3FE]|[U0001F9AF-U0001F9B3U0001F9BCU0001F9BD]))?)?|U0001F469(?:u200D(?:[u2695u2696u2708]uFE0F?|u2764uFE0F?u200D(?:U0001F48Bu200D)?[U0001F468U0001F469]|[U0001F33EU0001F373U0001F37CU0001F393U0001F3A4U0001F3A8U0001F3EBU0001F3ED]|U0001F466(?:u200DU0001F466)?|U0001F467(?:u200D[U0001F466U0001F467])?|U0001F469u200D(?:U0001F466(?:u200DU0001F466)?|U0001F467(?:u200D[U0001F466U0001F467])?)|[U0001F4BBU0001F4BCU0001F527U0001F52CU0001F680U0001F692U0001F9AF-U0001F9B3U0001F9BCU0001F9BD])|U0001F3FB(?:u200D(?:[u2695u2696u2708]uFE0F?|u2764uFE0F?u200D(?:[U0001F468U0001F469][U0001F3FB-U0001F3FF]|U0001F48Bu200D[U0001F468U0001F469][U0001F3FB-U0001F3FF])|[U0001F33EU0001F373U0001F37CU0001F393U0001F3A4U0001F3A8U0001F3EBU0001F3EDU0001F4BBU0001F4BCU0001F527U0001F52CU0001F680U0001F692]|U0001F91Du200D[U0001F468U0001F469][U0001F3FC-U0001F3FF]|[U0001F9AF-U0001F9B3U0001F9BCU0001F9BD]))?|U0001F3FC(?:u200D(?:[u2695u2696u2708]uFE0F?|u2764uFE0F?u200D(?:[U0001F468U0001F469][U0001F3FB-U0001F3FF]|U0001F48Bu200D[U0001F468U0001F469][U0001F3FB-U0001F3FF])|[U0001F33EU0001F373U0001F37CU0001F393U0001F3A4U0001F3A8U0001F3EBU0001F3EDU0001F4BBU0001F4BCU0001F527U0001F52CU0001F680U0001F692]|U0001F91Du200D[U0001F468U0001F469][U0001F3FBU0001F3FD-U0001F3FF]|[U0001F9AF-U0001F9B3U0001F9BCU0001F9BD]))?|U0001F3FD(?:u200D(?:[u2695u2696u2708]uFE0F?|u2764uFE0F?u200D(?:[U0001F468U0001F469][U0001F3FB-U0001F3FF]|U0001F48Bu200D[U0001F468U0001F469][U0001F3FB-U0001F3FF])|[U0001F33EU0001F373U0001F37CU0001F393U0001F3A4U0001F3A8U0001F3EBU0001F3EDU0001F4BBU0001F4BCU0001F527U0001F52CU0001F680U0001F692]|U0001F91Du200D[U0001F468U0001F469][U0001F3FBU0001F3FCU0001F3FEU0001F3FF]|[U0001F9AF-U0001F9B3U0001F9BCU0001F9BD]))?|U0001F3FE(?:u200D(?:[u2695u2696u2708]uFE0F?|u2764uFE0F?u200D(?:[U0001F468U0001F469][U0001F3FB-U0001F3FF]|U0001F48Bu200D[U0001F468U0001F469][U0001F3FB-U0001F3FF])|[U0001F33EU0001F373U0001F37CU0001F393U0001F3A4U0001F3A8U0001F3EBU0001F3EDU0001F4BBU0001F4BCU0001F527U0001F52CU0001F680U0001F692]|U0001F91Du200D[U0001F468U0001F469][U0001F3FB-U0001F3FDU0001F3FF]|[U0001F9AF-U0001F9B3U0001F9BCU0001F9BD]))?|U0001F3FF(?:u200D(?:[u2695u2696u2708]uFE0F?|u2764uFE0F?u200D(?:[U0001F468U0001F469][U0001F3FB-U0001F3FF]|U0001F48Bu200D[U0001F468U0001F469][U0001F3FB-U0001F3FF])|[U0001F33EU0001F373U0001F37CU0001F393U0001F3A4U0001F3A8U0001F3EBU0001F3EDU0001F4BBU0001F4BCU0001F527U0001F52CU0001F680U0001F692]|U0001F91Du200D[U0001F468U0001F469][U0001F3FB-U0001F3FE]|[U0001F9AF-U0001F9B3U0001F9BCU0001F9BD]))?)?|U0001F46A|[U0001F46B-U0001F46D][U0001F3FB-U0001F3FF]?|U0001F46E(?:u200D[u2640u2642]uFE0F?|[U0001F3FB-U0001F3FF](?:u200D[u2640u2642]uFE0F?)?)?|U0001F46F(?:u200D[u2640u2642]uFE0F?)?|[U0001F470U0001F471](?:u200D[u2640u2642]uFE0F?|[U0001F3FB-U0001F3FF](?:u200D[u2640u2642]uFE0F?)?)?|U0001F472[U0001F3FB-U0001F3FF]?|U0001F473(?:u200D[u2640u2642]uFE0F?|[U0001F3FB-U0001F3FF](?:u200D[u2640u2642]uFE0F?)?)?|[U0001F474-U0001F476][U0001F3FB-U0001F3FF]?|U0001F477(?:u200D[u2640u2642]uFE0F?|[U0001F3FB-U0001F3FF](?:u200D[u2640u2642]uFE0F?)?)?|U0001F478[U0001F3FB-U0001F3FF]?|[U0001F479-U0001F47B]|U0001F47C[U0001F3FB-U0001F3FF]?|[U0001F47D-U0001F480]|[U0001F481U0001F482](?:u200D[u2640u2642]uFE0F?|[U0001F3FB-U0001F3FF](?:u200D[u2640u2642]uFE0F?)?)?|U0001F483[U0001F3FB-U0001F3FF]?|U0001F484|U0001F485[U0001F3FB-U0001F3FF]?|[U0001F486U0001F487](?:u200D[u2640u2642]uFE0F?|[U0001F3FB-U0001F3FF](?:u200D[u2640u2642]uFE0F?)?)?|[U0001F488-U0001F48E]|U0001F48F[U0001F3FB-U0001F3FF]?|U0001F490|U0001F491[U0001F3FB-U0001F3FF]?|[U0001F492-U0001F4A9]|U0001F4AA[U0001F3FB-U0001F3FF]?|[U0001F4AB-U0001F4FC]|U0001F4FDuFE0F?|[U0001F4FF-U0001F53D]|[U0001F549U0001F54A]uFE0F?|[U0001F54B-U0001F54EU0001F550-U0001F567]|[U0001F56FU0001F570U0001F573]uFE0F?|U0001F574[uFE0FU0001F3FB-U0001F3FF]?|U0001F575(?:u200D[u2640u2642]uFE0F?|[uFE0FU0001F3FB-U0001F3FF](?:u200D[u2640u2642]uFE0F?)?)?|[U0001F576-U0001F579]uFE0F?|U0001F57A[U0001F3FB-U0001F3FF]?|[U0001F587U0001F58A-U0001F58D]uFE0F?|U0001F590[uFE0FU0001F3FB-U0001F3FF]?|[U0001F595U0001F596][U0001F3FB-U0001F3FF]?|U0001F5A4|[U0001F5A5U0001F5A8U0001F5B1U0001F5B2U0001F5BCU0001F5C2-U0001F5C4U0001F5D1-U0001F5D3U0001F5DC-U0001F5DEU00

与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
OGeek|极客中国-欢迎来到极客的世界,一个免费开放的程序员编程交流平台!开放,进步,分享!让技术改变生活,让极客改变未来! Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Click Here to Ask a Question

1.4m articles

1.4m replys

5 comments

56.9k users

...