Normal closure of a non-normal subgroup

When $G$ is a finite group and $H$ is a proper subgroup, it is well-known that the union of all the conjugate subgroups $\bigcup_{g\in G} gHg^{-1}$ is never equal to $G$. (The same is true when finiteness of $G$ is weakened to the index $[G:H]$ being finite.) When $H \lhd G$, this union is equal to $H$ and thus is a subgroup. Can this union be a subgroup of $G$ when $H \not\lhd G$?

Equivalently, since the normal closure of $H$ in $G$ is the subgroup of $G$ generated by $\bigcup_{g\in G} gHg^{-1}$, can the normal closure of $H$ in $G$ ever just be that union when $H\not\lhd G$?

I have been unable to find any examples of this or find the issue discussed anywhere. Its relation to the normal closure of $H$ reminds me of the "reverse" question whether the commutator subgroup is ever larger than the set of all commutators, except I have seen that commutator subgroup question discussed in group theory books and examples are easily found online (with the smallest one having order $96$).

A family of examples is given by Keith Kearnes in a comment, which I will put here as an answer.

Let $V$ be a finite-dimensional $\mathbf F_p$-vector space with dimension $d \geq 1$. Set $G = V \rtimes {\rm Aut}(V)$, with group law $$ (v,A)(w,B) = (v + A(w),AB) $$ and $(v,A)^{-1} = (-A^{-1}(v),A^{-1})$.

Theorem. With notation as above, let $L$ be a $1$-dimensional subspace of $V$ and $H = (L,1)$. Then $H$ is a subgroup of $G$ and $$ \bigcup_{g \in G} gHg^{-1} = (V,1) = \{(v,1) : v \in V\}, $$ which is a subgroup of $G$ and is strictly larger than $H$ when $d > 1$.

Proof. Let $v_0$ be a nonzero vector in $L$, so $L = \mathbf F_pv_0$ and $H = \langle (v_0,1)\rangle$. For $g = (v,A) \in G$, $$ g(v_0,1)g^{-1} = (A(v_0),1), $$ where the $v$ in $g$ plays no role in the end result. Thus $gHg^{-1} = g\langle(v_0,1)\rangle{g}^{-1} = \langle (A(v_0),1)\rangle$. Since ${\rm Aut}(V) = {\rm GL}(V)$, $\bigcup_{g \in G} g(v_0,1)g^{-1} = \bigcup_{A \in {\rm GL}(V)} A(v_0) = V - \{0\}$ and $(0,1) \in H$, so $$ \bigcup_{g \in G} gHg^{-1} = (V,1), $$ which is a subgroup of $G$. It is strictly larger than $H$ unless $L = V$, which means $V$ is $1$-dimensional. So as long as $V$ has dimension greater than $1$, $\bigcup_{g \in G} gHg^{-1} \not= H$. QED

In the proof, we relied on the property that ${\rm Aut}(V)$ acts transitively on $V - \{0\}$. Every nontrivial finite group with that property is a finite-dimensional vector space over $\mathbf F_p$ for some prime $p$: see Theorem $3.15$ here.

One way to think about $V \rtimes {\rm Aut}(V)$ is as the set of affine-linear maps $V \to V$, where $(v,A)$ in the semi-direct product corresponds to the affine-linear map $\mathbf x \mapsto A\mathbf x + v$ on $V$. The smallest example of such $V$ is $\mathbf F_2^2$, for which $V \rtimes {\rm Aut}(V)$ has size $4 \cdot 6 = 24$ and the group of affine-linear maps $\mathbf F_2^2 \to \mathbf F_2^2$ is isomorphic to the group $S_4$. So $S_4$ is an example of the kind of group I was seeking, with $\mathbf F_2^2$ embedded in $S_4$ as the translations on $\mathbf F_2^2$, where the nonzero translations are the permutations of type $(2,2)$. That leads to a very concrete example.

Example. Let $G = S_4$ and $H = \langle (12)(34)\rangle$. Then $$ \bigcup_{g \in G} gHg^{-1} = \{(1),(12)(34),(13)(24),(14)(23)\} $$ is the normal subgroup of $S_4$ with order $4$.